[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:53 dist-test-slave-2m5f krb5kdc[5601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:53 dist-test-slave-2m5f krb5kdc[5601](info): set up 2 sockets
Jun 20 14:50:53 dist-test-slave-2m5f krb5kdc[5601](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:50:54.916656 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:50:55.044260 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:34423
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:34423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:55.127008 5614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:55.127285 5614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:55.127326 5614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:55.128583 5614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:55.128665 5614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:55.128688 5614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:55.128710 5614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:55.128738 5614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:55.130504 5614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:34423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:34423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:55.130828 5614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:55.131111 5614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:55.137888 5622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:55.137888 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:55.138106 5614 server_base.cc:1034] running on GCE node
W20260620 14:50:55.139706 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:55.140225 5614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:55.140676 5614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:55.141870 5614 hybrid_clock.cc:648] HybridClock initialized: now 1781967055141822 us; error 66 us; skew 500 ppm
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:55.145202 5614 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:50:55.145709 5614 webserver.cc:460] Webserver started at http://127.5.118.190:45879/ using document root <none> and password file <none>
I20260620 14:50:55.146013 5614 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:55.146063 5614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:55.146194 5614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:55.147449 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "820ce196113343a08c5a563c5ee08a58"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "aedfd75cab25de5f9d54ce102278661d"
server_key_iv: "dc9aeae0632fe9a05a9366b37082b666"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.147838 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "820ce196113343a08c5a563c5ee08a58"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "aedfd75cab25de5f9d54ce102278661d"
server_key_iv: "dc9aeae0632fe9a05a9366b37082b666"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.149957 5614 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 14:50:55.151046 5629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:55.151314 5614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:50:55.151415 5614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "820ce196113343a08c5a563c5ee08a58"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "aedfd75cab25de5f9d54ce102278661d"
server_key_iv: "dc9aeae0632fe9a05a9366b37082b666"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.151506 5614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:55.190898 5614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:55.193153 5614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:55.193280 5614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:55.197281 5614 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:34423
I20260620 14:50:55.197319 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:34423 every 8 connection(s)
I20260620 14:50:55.197736 5614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:55.198676 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:55.200027 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 5614
I20260620 14:50:55.200163 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:50:55.200829 5594 external_mini_cluster.cc:1428] Setting key 84f5fd76810ff475b77ee43a08524c37
I20260620 14:50:55.201453 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58: Bootstrap starting.
I20260620 14:50:55.202236 5682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:55.202638 5682 log.cc:824] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:55.203471 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58: No bootstrap required, opened a new log
I20260620 14:50:55.205718 5682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820ce196113343a08c5a563c5ee08a58" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34423 } }
I20260620 14:50:55.205868 5682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:55.205891 5682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820ce196113343a08c5a563c5ee08a58, State: Initialized, Role: FOLLOWER
I20260620 14:50:55.206156 5682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820ce196113343a08c5a563c5ee08a58" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34423 } }
I20260620 14:50:55.206255 5682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:55.206305 5682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:55.206363 5682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:55.207330 5682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820ce196113343a08c5a563c5ee08a58" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34423 } }
I20260620 14:50:55.207468 5682 leader_election.cc:302] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 820ce196113343a08c5a563c5ee08a58; no voters:
I20260620 14:50:55.207671 5682 leader_election.cc:288] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:55.207700 5687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:55.207839 5687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [term 1 LEADER]: Becoming Leader. State: Replica: 820ce196113343a08c5a563c5ee08a58, State: Running, Role: LEADER
I20260620 14:50:55.207988 5682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:55.208014 5687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820ce196113343a08c5a563c5ee08a58" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34423 } }
I20260620 14:50:55.208521 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "820ce196113343a08c5a563c5ee08a58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820ce196113343a08c5a563c5ee08a58" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34423 } } }
I20260620 14:50:55.208616 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:55.208547 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 820ce196113343a08c5a563c5ee08a58. Latest consensus state: current_term: 1 leader_uuid: "820ce196113343a08c5a563c5ee08a58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "820ce196113343a08c5a563c5ee08a58" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34423 } } }
I20260620 14:50:55.208662 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:55.208962 5694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:55.209581 5694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:55.211272 5694 catalog_manager.cc:1347] Generated new cluster ID: b65a4afe1d5542899e4d0e37ffd7b629
I20260620 14:50:55.211339 5694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:55.220090 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.201669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.5.118.190:34423)
0620 14:50:55.201878 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:55.201888 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:55.202176 (+ 288us) server_negotiation.cc:407] Connection header received
0620 14:50:55.205722 (+ 3546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:55.205730 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:55.205784 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:55.206042 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:55.207831 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.209183 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.210875 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.211524 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.211667 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:55.215541 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:55.215546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:55.215546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:55.215558 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:55.218003 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.218485 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.218489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.218489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.218598 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.218893 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.218894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.218895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.219174 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:55.219355 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:55.219673 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 14:50:55.219827 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260620 14:50:55.230443 5694 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:55.231104 5694 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:55.236034 5694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 820ce196113343a08c5a563c5ee08a58: Generated new TSK 0
I20260620 14:50:55.236308 5694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:50:55.269475 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34423
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:55.373564 5708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:55.373785 5708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:55.373809 5708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:55.375078 5708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:55.375123 5708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:55.375152 5708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:50:55.376753 5708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34423
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:55.377014 5708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:55.377262 5708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:55.382488 5713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:55.382481 5714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:55.382509 5716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:55.382717 5708 server_base.cc:1034] running on GCE node
I20260620 14:50:55.382942 5708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:55.383199 5708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:55.384377 5708 hybrid_clock.cc:648] HybridClock initialized: now 1781967055384367 us; error 36 us; skew 500 ppm
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:55.387074 5708 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:50:55.387576 5708 webserver.cc:460] Webserver started at http://127.5.118.129:36925/ using document root <none> and password file <none>
I20260620 14:50:55.387813 5708 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:55.387881 5708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:55.388026 5708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:55.389195 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "ae8afd69d39a43138ada2ea57c83e9a5"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "b6d502133bf443a4b1c681800b4c47fa"
server_key_iv: "02801fe49572b44371199ed18e8c2c1d"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.389575 5708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "ae8afd69d39a43138ada2ea57c83e9a5"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "b6d502133bf443a4b1c681800b4c47fa"
server_key_iv: "02801fe49572b44371199ed18e8c2c1d"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.390939 5708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:55.391722 5723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:55.391906 5708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:50:55.392005 5708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "ae8afd69d39a43138ada2ea57c83e9a5"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "b6d502133bf443a4b1c681800b4c47fa"
server_key_iv: "02801fe49572b44371199ed18e8c2c1d"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.392086 5708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:55.412529 5708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:55.414657 5708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:55.414841 5708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:55.415100 5708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:55.415444 5708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:55.415503 5708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:55.415540 5708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:55.415585 5708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:55.421403 5708 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:33521
I20260620 14:50:55.421442 5836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:33521 every 8 connection(s)
I20260620 14:50:55.421792 5708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:55.425673 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 5708
I20260620 14:50:55.425791 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:55.425961 5594 external_mini_cluster.cc:1428] Setting key 9cff283911de698e9becabaa21666dd0
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:55.433110 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.422666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:59449 (local address 127.5.118.190:34423)
0620 14:50:55.422751 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:55.422754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:55.422978 (+ 224us) server_negotiation.cc:407] Connection header received
0620 14:50:55.423601 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:55.423602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:55.423622 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:55.423680 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:55.424835 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.425408 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.427062 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.427267 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.427330 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:55.429945 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:55.429947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:55.429947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:55.429960 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:55.431511 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.431971 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.431973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.431973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.432041 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.432363 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.432364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.432364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.432486 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:55.432573 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:55.432930 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 14:50:55.433019 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:50:55.433243 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.422539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:34423 (local address 127.5.118.129:59449)
0620 14:50:55.422777 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:55.422873 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:55.423471 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:55.423762 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:55.423766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:55.424493 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:55.424712 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:55.424718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.425574 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.425575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:55.426978 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:55.426982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.427389 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.427390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:55.427565 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:55.428260 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:55.428273 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:55.429804 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:55.431628 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:55.431638 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:55.431640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:55.431881 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:55.432109 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:55.432109 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:55.432110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:55.432280 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:55.432552 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:55.432558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:55.432711 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 14:50:55.432812 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:50:55.432998 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
I20260620 14:50:55.433694 5837 heartbeater.cc:347] Connected to a master server at 127.5.118.190:34423
I20260620 14:50:55.433790 5837 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
I20260620 14:50:55.434003 5837 heartbeater.cc:511] Master 127.5.118.190:34423 requested a full tablet report, sending...
I20260620 14:50:55.434507 5646 ts_manager.cc:194] Registered new tserver with Master: ae8afd69d39a43138ada2ea57c83e9a5 (127.5.118.129:33521)
I20260620 14:50:55.435099 5646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:59449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:50:55.451856 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34423
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:55.536254 5842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:55.536490 5842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:55.536517 5842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:55.537694 5842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:55.537735 5842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:55.537762 5842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:50:55.539156 5842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34423
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:55.539357 5842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:55.539537 5842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:50:55.544790 5842 server_base.cc:1034] running on GCE node
W20260620 14:50:55.544683 5850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:55.544677 5847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:55.544683 5848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:55.545218 5842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:55.545436 5842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:55.546583 5842 hybrid_clock.cc:648] HybridClock initialized: now 1781967055546567 us; error 36 us; skew 500 ppm
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:55.549124 5842 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:50:55.549631 5842 webserver.cc:460] Webserver started at http://127.5.118.130:40551/ using document root <none> and password file <none>
I20260620 14:50:55.549860 5842 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:55.549929 5842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:55.550074 5842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:55.551153 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "37fd2782b9ba4468823f7d74cb01e1e5"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "a6a1dcf1976a70f07eee64e00ba80998"
server_key_iv: "095504fbabadcb05f9b903deadd0a6bc"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.551577 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "37fd2782b9ba4468823f7d74cb01e1e5"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "a6a1dcf1976a70f07eee64e00ba80998"
server_key_iv: "095504fbabadcb05f9b903deadd0a6bc"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.553138 5842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:50:55.553856 5857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:55.554028 5842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:55.554127 5842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "37fd2782b9ba4468823f7d74cb01e1e5"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "a6a1dcf1976a70f07eee64e00ba80998"
server_key_iv: "095504fbabadcb05f9b903deadd0a6bc"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.554198 5842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:55.561749 5842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:55.563747 5842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:55.563910 5842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:55.564139 5842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:55.564479 5842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:55.564531 5842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:55.564572 5842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:55.564599 5842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:55.570936 5842 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:44009
I20260620 14:50:55.570989 5970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:44009 every 8 connection(s)
I20260620 14:50:55.571350 5842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:55.576962 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 5842
I20260620 14:50:55.577080 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:55.577246 5594 external_mini_cluster.cc:1428] Setting key 8c8bf6dbbd405ada54c44eca218223b2
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:55.582145 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.572131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:43189 (local address 127.5.118.190:34423)
0620 14:50:55.572204 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:55.572207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:55.572397 (+ 190us) server_negotiation.cc:407] Connection header received
0620 14:50:55.572880 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:55.572881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:55.572892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:55.572957 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:55.573969 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.574470 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.575592 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.575801 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.575866 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:55.578459 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:55.578460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:55.578461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:55.578469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:55.580625 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.581123 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.581125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.581126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.581196 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.581481 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.581482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.581483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.581603 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:55.581677 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:55.581962 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:50:55.582050 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:50:55.582366 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.572090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:34423 (local address 127.5.118.130:43189)
0620 14:50:55.572245 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:55.572289 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:55.572771 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:55.573035 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:55.573039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:55.573608 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:55.573878 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:55.573884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.574583 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.574584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:55.575508 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:55.575511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.575924 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.575925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:55.576051 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:55.576604 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:55.576613 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:55.578366 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:55.580701 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:55.580708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:55.580710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:55.581015 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:55.581294 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:55.581295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:55.581295 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:55.581408 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:55.581667 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:55.581673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:55.581823 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 14:50:55.581943 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:50:55.582109 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":101,"threads_started":1}
I20260620 14:50:55.582890 5971 heartbeater.cc:347] Connected to a master server at 127.5.118.190:34423
I20260620 14:50:55.583009 5971 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:55.583227 5971 heartbeater.cc:511] Master 127.5.118.190:34423 requested a full tablet report, sending...
I20260620 14:50:55.583667 5646 ts_manager.cc:194] Registered new tserver with Master: 37fd2782b9ba4468823f7d74cb01e1e5 (127.5.118.130:44009)
I20260620 14:50:55.584146 5646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:43189
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:50:55.607926 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34423
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:55.696365 5976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:55.696581 5976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:55.696607 5976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:55.697757 5976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:55.697803 5976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:55.697832 5976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:50:55.699252 5976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34423
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:55.699462 5976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:55.699646 5976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:55.705044 5981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:55.705044 5982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:55.705081 5984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:55.705358 5976 server_base.cc:1034] running on GCE node
I20260620 14:50:55.705590 5976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:55.705852 5976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:55.707029 5976 hybrid_clock.cc:648] HybridClock initialized: now 1781967055707012 us; error 35 us; skew 500 ppm
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:55.710233 5976 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:50:55.710703 5976 webserver.cc:460] Webserver started at http://127.5.118.131:34635/ using document root <none> and password file <none>
I20260620 14:50:55.711015 5976 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:55.711086 5976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:55.711237 5976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:55.712410 5976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "2608d3d1585444a2aabff9360f1cc19e"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "97a0d479afcf1cd4905d26188c14c28d"
server_key_iv: "4396918e8d34582b927182cc6576689d"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.712798 5976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "2608d3d1585444a2aabff9360f1cc19e"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "97a0d479afcf1cd4905d26188c14c28d"
server_key_iv: "4396918e8d34582b927182cc6576689d"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.714241 5976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:55.715170 5991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:55.715342 5976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:50:55.715449 5976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "2608d3d1585444a2aabff9360f1cc19e"
format_stamp: "Formatted at 2026-06-20 14:50:55 on dist-test-slave-2m5f"
server_key: "97a0d479afcf1cd4905d26188c14c28d"
server_key_iv: "4396918e8d34582b927182cc6576689d"
server_key_version: "encryptionkey@0"
I20260620 14:50:55.715524 5976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:55.734952 5976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:55.737550 5976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:55.737731 5976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:55.738001 5976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:55.738343 5976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:55.738404 5976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:55.738440 5976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:55.738466 5976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:55.745021 5976 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:40341
I20260620 14:50:55.745059 6104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:40341 every 8 connection(s)
I20260620 14:50:55.745397 5976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:55.753921 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 5976
I20260620 14:50:55.754019 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:55.754215 5594 external_mini_cluster.cc:1428] Setting key bd8afe5385e536feba770c32a63ee8a7
I20260620 14:50:55.755308 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.746158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:50301 (local address 127.5.118.190:34423)
0620 14:50:55.746232 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:55.746242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:55.746433 (+ 191us) server_negotiation.cc:407] Connection header received
0620 14:50:55.746943 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:55.746945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:55.746956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:55.747018 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:55.748023 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.748623 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.749696 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.749899 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.749962 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:55.752330 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:55.752332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:55.752332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:55.752342 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:55.753602 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.754174 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.754176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.754177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.754243 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.754542 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.754543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.754543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.754688 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:55.754756 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:55.755126 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 14:50:55.755210 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:50:55.755584 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.746122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:34423 (local address 127.5.118.131:50301)
0620 14:50:55.746306 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:55.746349 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:55.746812 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:55.747093 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:55.747097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:55.747708 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:55.747917 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:55.747922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.748728 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.748729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:55.749607 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:55.749609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.749968 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.749969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:55.750076 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:55.750590 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:55.750599 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:55.752225 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:55.753800 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:55.753806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:55.753808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:55.754082 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:55.754325 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:55.754326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:55.754326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:55.754456 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:55.754762 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:55.754769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:55.754953 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 14:50:55.755090 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 14:50:55.755335 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260620 14:50:55.756160 6105 heartbeater.cc:347] Connected to a master server at 127.5.118.190:34423
I20260620 14:50:55.756351 6105 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:55.756574 6105 heartbeater.cc:511] Master 127.5.118.190:34423 requested a full tablet report, sending...
I20260620 14:50:55.756968 5646 ts_manager.cc:194] Registered new tserver with Master: 2608d3d1585444a2aabff9360f1cc19e (127.5.118.131:40341)
I20260620 14:50:55.757347 5646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:50301
I20260620 14:50:55.766721 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.118.129@KRBTEST.COM
I20260620 14:50:55.791616 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.783101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.5.118.129:33521)
0620 14:50:55.783302 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:55.783307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:55.783319 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:50:55.783348 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:55.783350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:55.783364 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:55.783432 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:55.784051 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.784985 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.785728 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.786131 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.786194 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:55.788606 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:55.788615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:55.788615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:55.788623 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:55.790362 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.790727 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.790731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.790731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.790866 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.791070 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.791070 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.791071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.791218 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:55.791352 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:55.791415 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:50:55.791487 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
W20260620 14:50:55.792066 5751 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34612
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.129@KRBTEST.COM
I20260620 14:50:55.815224 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.807733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.5.118.129:33521)
0620 14:50:55.807902 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:55.807904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:55.807914 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:50:55.807940 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:55.807941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:55.807952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:55.808032 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:55.808753 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.809381 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.810154 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:55.810392 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:55.810460 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:55.812781 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:55.812782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:55.812782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:55.812795 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:55.814099 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.814441 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.814442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.814442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.814496 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:55.814661 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:55.814661 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:55.814662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:55.814753 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:55.814911 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:55.815055 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:50:55.815122 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:50:55.815923 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 5708
I20260620 14:50:55.821105 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 5842
I20260620 14:50:55.826722 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 5976
I20260620 14:50:55.831395 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 5614
2026-06-20T14:50:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1973 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[6119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[6119](info): set up 2 sockets
Jun 20 14:50:55 dist-test-slave-2m5f krb5kdc[6119](info): commencing operation
krb5kdc: starting...
W20260620 14:50:56.952534 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:50:57.079768 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:36835
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:36835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:57.162135 6132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:57.162412 6132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:57.162458 6132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:57.163671 6132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:57.163748 6132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:57.163772 6132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:57.163794 6132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:57.163821 6132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:57.165369 6132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:36835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:36835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:57.165660 6132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:57.165894 6132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:57.171767 6138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:57.171949 6132 server_base.cc:1034] running on GCE node
W20260620 14:50:57.171830 6137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:57.171767 6140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:57.172389 6132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:57.172731 6132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:57.173914 6132 hybrid_clock.cc:648] HybridClock initialized: now 1781967057173902 us; error 38 us; skew 500 ppm
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:57.176765 6132 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:50:57.177341 6132 webserver.cc:460] Webserver started at http://127.5.118.190:38229/ using document root <none> and password file <none>
I20260620 14:50:57.177587 6132 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:57.177655 6132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:57.177810 6132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:57.178982 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "20483bb6450a4186900d088469b32849"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "224107550bdc8188302871e7fbd6356e"
server_key_iv: "8c7cad0df035922159f358cb99ac1681"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.179373 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "20483bb6450a4186900d088469b32849"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "224107550bdc8188302871e7fbd6356e"
server_key_iv: "8c7cad0df035922159f358cb99ac1681"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.181016 6132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:57.181968 6147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:57.182164 6132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:50:57.182276 6132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "20483bb6450a4186900d088469b32849"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "224107550bdc8188302871e7fbd6356e"
server_key_iv: "8c7cad0df035922159f358cb99ac1681"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.182367 6132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:57.208477 6132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:57.212082 6132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:57.212330 6132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:57.216799 6132 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:36835
I20260620 14:50:57.217204 6199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:36835 every 8 connection(s)
I20260620 14:50:57.217263 6132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:57.218055 6200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:57.220595 6200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849: Bootstrap starting.
I20260620 14:50:57.221159 6200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:57.221418 6200 log.cc:824] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:57.222127 6200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849: No bootstrap required, opened a new log
I20260620 14:50:57.223775 6200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20483bb6450a4186900d088469b32849" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36835 } }
I20260620 14:50:57.223935 6200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:57.223958 6200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20483bb6450a4186900d088469b32849, State: Initialized, Role: FOLLOWER
I20260620 14:50:57.224109 6200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20483bb6450a4186900d088469b32849" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36835 } }
I20260620 14:50:57.224227 6200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:57.224273 6200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:57.224309 6200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:57.224893 6200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20483bb6450a4186900d088469b32849" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36835 } }
I20260620 14:50:57.225024 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 6132
I20260620 14:50:57.225026 6200 leader_election.cc:302] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20483bb6450a4186900d088469b32849; no voters:
I20260620 14:50:57.225096 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:50:57.225230 6200 leader_election.cc:288] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:57.225292 5594 external_mini_cluster.cc:1428] Setting key 086b2d7f21f6aba21a025bcdd1fc1f44
I20260620 14:50:57.225308 6203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:57.225546 6200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:57.225705 6203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [term 1 LEADER]: Becoming Leader. State: Replica: 20483bb6450a4186900d088469b32849, State: Running, Role: LEADER
I20260620 14:50:57.225822 6203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20483bb6450a4186900d088469b32849" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36835 } }
I20260620 14:50:57.226517 6205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20483bb6450a4186900d088469b32849. Latest consensus state: current_term: 1 leader_uuid: "20483bb6450a4186900d088469b32849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20483bb6450a4186900d088469b32849" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36835 } } }
I20260620 14:50:57.226611 6205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:57.226673 6204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20483bb6450a4186900d088469b32849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20483bb6450a4186900d088469b32849" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36835 } } }
I20260620 14:50:57.226748 6204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849 [sys.catalog]: This master's current role is: LEADER
W20260620 14:50:57.227656 6220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:57.227721 6220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:57.227880 6216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:57.228108 6216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:57.231762 6216 catalog_manager.cc:1347] Generated new cluster ID: 5500f62fa3ea4b32b7609cce513a1d67
I20260620 14:50:57.231837 6216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:57.237973 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.226010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.5.118.190:36835)
0620 14:50:57.226474 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.226485 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.226524 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:50:57.226674 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.226683 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.226743 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.227840 (+ 1097us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:57.227959 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.229447 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.230313 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.231028 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.231153 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.234093 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:57.234098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:57.234099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:57.234115 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:57.236041 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.236528 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.236533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.236533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.236644 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.236872 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.236873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.236874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.237186 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:57.237375 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.237498 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.237771 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":85,"threads_started":1}
I20260620 14:50:57.241200 6216 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:57.241925 6216 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:57.249097 6216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20483bb6450a4186900d088469b32849: Generated new TSK 0
I20260620 14:50:57.249328 6216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:50:57.284394 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36835
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:57.372691 6226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:57.372962 6226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:57.373008 6226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:57.374343 6226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:57.374439 6226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:57.374490 6226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:50:57.376224 6226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:57.376516 6226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:57.376771 6226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:57.382681 6232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:57.382669 6234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:57.382669 6231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:57.383016 6226 server_base.cc:1034] running on GCE node
I20260620 14:50:57.383242 6226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:57.383466 6226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:57.384613 6226 hybrid_clock.cc:648] HybridClock initialized: now 1781967057384598 us; error 28 us; skew 500 ppm
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:57.387480 6226 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:50:57.387967 6226 webserver.cc:460] Webserver started at http://127.5.118.129:40707/ using document root <none> and password file <none>
I20260620 14:50:57.388249 6226 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:57.388319 6226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:57.388479 6226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:57.389575 6226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "3a761811e6284230a64a7236312667f1"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "4138d44fd0c7b137baae72da4cb02d1e"
server_key_iv: "7991c1c02521caf66a9e6b52913b6a3f"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.389969 6226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "3a761811e6284230a64a7236312667f1"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "4138d44fd0c7b137baae72da4cb02d1e"
server_key_iv: "7991c1c02521caf66a9e6b52913b6a3f"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.391305 6226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:50:57.392035 6241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:57.392256 6226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:50:57.392385 6226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "3a761811e6284230a64a7236312667f1"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "4138d44fd0c7b137baae72da4cb02d1e"
server_key_iv: "7991c1c02521caf66a9e6b52913b6a3f"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.392459 6226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:57.398865 6226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:57.400837 6226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:57.400990 6226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:57.401204 6226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:57.401511 6226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:57.401563 6226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:57.401607 6226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:57.401643 6226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:57.407498 6226 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:46591
I20260620 14:50:57.407603 6354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:46591 every 8 connection(s)
I20260620 14:50:57.407896 6226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:57.410279 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 6226
I20260620 14:50:57.410380 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:57.410554 5594 external_mini_cluster.cc:1428] Setting key 6b12fe65faed9b1d908458f0669a0734
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
I20260620 14:50:57.419770 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.408725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:37935 (local address 127.5.118.190:36835)
0620 14:50:57.408776 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.408778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.409090 (+ 312us) server_negotiation.cc:407] Connection header received
0620 14:50:57.410162 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.410164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.410179 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.410293 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:57.411198 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.411722 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.413124 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.413510 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.413596 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.416426 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:57.416428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:57.416429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:57.416439 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:57.418215 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.418681 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.418682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.418683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.418750 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.419011 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.419012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.419013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.419165 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:57.419246 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.419574 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.419667 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:50:57.420038 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.408706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:36835 (local address 127.5.118.129:37935)
0620 14:50:57.408966 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.409004 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:57.409924 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:57.410459 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:57.410464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:57.410839 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:57.411101 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.411108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.411827 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.411828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.413022 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.413025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.413619 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.413621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.413736 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.414356 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:57.414366 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:57.416301 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:57.418347 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:57.418354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:57.418356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:57.418598 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:57.418828 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:57.418829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:57.418830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:57.418942 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:57.419238 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:57.419245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:57.419375 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 14:50:57.419542 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 14:50:57.419684 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":138,"threads_started":1}
I20260620 14:50:57.420693 6355 heartbeater.cc:347] Connected to a master server at 127.5.118.190:36835
I20260620 14:50:57.420799 6355 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:57.420959 6355 heartbeater.cc:511] Master 127.5.118.190:36835 requested a full tablet report, sending...
I20260620 14:50:57.421546 6164 ts_manager.cc:194] Registered new tserver with Master: 3a761811e6284230a64a7236312667f1 (127.5.118.129:46591)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
I20260620 14:50:57.422405 6164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:37935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:50:57.439453 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36835
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:57.527154 6360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:57.527398 6360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:57.527439 6360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:57.528676 6360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:57.528745 6360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:57.528790 6360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:50:57.530341 6360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:57.530618 6360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:57.530874 6360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:57.536041 6366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:57.536194 6365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:57.536077 6368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:57.536322 6360 server_base.cc:1034] running on GCE node
I20260620 14:50:57.536566 6360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:57.536777 6360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:57.537937 6360 hybrid_clock.cc:648] HybridClock initialized: now 1781967057537915 us; error 37 us; skew 500 ppm
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:57.540597 6360 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:50:57.541092 6360 webserver.cc:460] Webserver started at http://127.5.118.130:40627/ using document root <none> and password file <none>
I20260620 14:50:57.541317 6360 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:57.541388 6360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:57.541536 6360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:57.542616 6360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "83aea54fa49e4120a45872d396721324"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "887e1c7d8c751c948bc7e700fef1c850"
server_key_iv: "3ec6cc0891c360c4af769a9e454493ea"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.542984 6360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "83aea54fa49e4120a45872d396721324"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "887e1c7d8c751c948bc7e700fef1c850"
server_key_iv: "3ec6cc0891c360c4af769a9e454493ea"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.544391 6360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:50:57.545159 6375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:57.545326 6360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:57.545449 6360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "83aea54fa49e4120a45872d396721324"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "887e1c7d8c751c948bc7e700fef1c850"
server_key_iv: "3ec6cc0891c360c4af769a9e454493ea"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.545522 6360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:57.566782 6360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:57.569301 6360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:57.569476 6360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:57.569720 6360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:57.570071 6360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:57.570125 6360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:57.570164 6360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:57.570192 6360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:57.576315 6360 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:45611
I20260620 14:50:57.576388 6488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:45611 every 8 connection(s)
I20260620 14:50:57.576722 6360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:57.584687 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 6360
I20260620 14:50:57.584792 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:57.584968 5594 external_mini_cluster.cc:1428] Setting key a2543657a65f36bea1edcd2ad4dbe27a
I20260620 14:50:57.586974 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.577504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:58353 (local address 127.5.118.190:36835)
0620 14:50:57.577575 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.577577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.577805 (+ 228us) server_negotiation.cc:407] Connection header received
0620 14:50:57.578896 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.578897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.578908 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.578972 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:57.579581 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.580085 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.581159 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.581359 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.581426 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.583773 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:57.583775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:57.583775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:57.583782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:57.585486 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.585988 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.585989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.585989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.586052 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.586315 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.586316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.586316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.586436 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:57.586507 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.586791 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.586885 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:50:57.587183 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.577481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:36835 (local address 127.5.118.130:58353)
0620 14:50:57.577630 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.577673 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:57.578751 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:57.579039 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:57.579042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:57.579299 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:57.579495 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.579499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.580167 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.580167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.581064 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.581066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.581440 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.581440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.581542 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.582190 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:57.582198 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:57.583660 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:57.585574 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:57.585582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:57.585584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:57.585895 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:57.586125 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:57.586125 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:57.586126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:57.586242 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:57.586509 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:57.586515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:57.586660 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 14:50:57.586760 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:50:57.586923 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260620 14:50:57.587618 6489 heartbeater.cc:347] Connected to a master server at 127.5.118.190:36835
I20260620 14:50:57.587721 6489 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:57.587908 6489 heartbeater.cc:511] Master 127.5.118.190:36835 requested a full tablet report, sending...
I20260620 14:50:57.588326 6164 ts_manager.cc:194] Registered new tserver with Master: 83aea54fa49e4120a45872d396721324 (127.5.118.130:45611)
I20260620 14:50:57.588708 6164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:58353
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:50:57.611913 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36835
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:36739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:57.697484 6494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:57.697698 6494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:57.697724 6494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:57.698936 6494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:57.698982 6494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:57.699011 6494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:50:57.700599 6494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:57.700866 6494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:57.701118 6494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:57.706463 6499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:57.706496 6494 server_base.cc:1034] running on GCE node
W20260620 14:50:57.706480 6502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:57.706518 6500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:57.707036 6494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:57.707312 6494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:57.708458 6494 hybrid_clock.cc:648] HybridClock initialized: now 1781967057708447 us; error 33 us; skew 500 ppm
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:57.711110 6494 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:50:57.711575 6494 webserver.cc:460] Webserver started at http://127.5.118.131:35435/ using document root <none> and password file <none>
I20260620 14:50:57.711774 6494 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:57.711818 6494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:57.711925 6494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:57.713014 6494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "ccbbc41c03594ae8b20e292da6c84ba2"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "3c3f5988326d8cb06a3c1f9bfc84ee88"
server_key_iv: "2ed9585177062ac8373572d8edf06835"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.713380 6494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "ccbbc41c03594ae8b20e292da6c84ba2"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "3c3f5988326d8cb06a3c1f9bfc84ee88"
server_key_iv: "2ed9585177062ac8373572d8edf06835"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.714717 6494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:50:57.715466 6509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:57.715636 6494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:57.715735 6494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "ccbbc41c03594ae8b20e292da6c84ba2"
format_stamp: "Formatted at 2026-06-20 14:50:57 on dist-test-slave-2m5f"
server_key: "3c3f5988326d8cb06a3c1f9bfc84ee88"
server_key_iv: "2ed9585177062ac8373572d8edf06835"
server_key_version: "encryptionkey@0"
I20260620 14:50:57.715816 6494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:57.739508 6494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:57.742097 6494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:57.742277 6494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:57.742556 6494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:57.742877 6494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:57.742929 6494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:57.742971 6494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:57.742996 6494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:57.749307 6494 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:45531
I20260620 14:50:57.749408 6622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:45531 every 8 connection(s)
I20260620 14:50:57.749732 6494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:57.757385 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 6494
I20260620 14:50:57.757494 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:57.757687 5594 external_mini_cluster.cc:1428] Setting key 161573a21847a69a401635b1d6aec4a2
I20260620 14:50:57.760066 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.750521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:53253 (local address 127.5.118.190:36835)
0620 14:50:57.750582 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.750583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.750889 (+ 306us) server_negotiation.cc:407] Connection header received
0620 14:50:57.751904 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.751906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.751918 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.751970 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:57.752709 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.753217 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.754336 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.754542 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.754625 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.756948 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:57.756949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:57.756950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:57.756957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:57.758622 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.759093 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.759094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.759094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.759150 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.759374 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.759375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.759375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.759510 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:57.759584 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.759915 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.759992 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:50:57.760284 6625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.750513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:36835 (local address 127.5.118.131:53253)
0620 14:50:57.750693 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.750740 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:57.751783 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:57.752047 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:57.752050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:57.752400 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:57.752606 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.752611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.753324 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.753325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.754218 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.754221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.754634 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.754635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.754764 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.755283 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:57.755291 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:57.756855 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:57.758718 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:57.758724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:57.758726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:57.759025 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:57.759197 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:57.759198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:57.759198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:57.759297 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:57.759586 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:57.759592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:57.759717 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 14:50:57.759873 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 14:50:57.760020 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260620 14:50:57.760787 6623 heartbeater.cc:347] Connected to a master server at 127.5.118.190:36835
I20260620 14:50:57.760893 6623 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:57.761111 6623 heartbeater.cc:511] Master 127.5.118.190:36835 requested a full tablet report, sending...
I20260620 14:50:57.761583 6164 ts_manager.cc:194] Registered new tserver with Master: ccbbc41c03594ae8b20e292da6c84ba2 (127.5.118.131:45531)
I20260620 14:50:57.762032 6164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:53253
I20260620 14:50:57.769793 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:57.777884 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.771579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.5.118.190:36835)
0620 14:50:57.771643 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.771645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.771732 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:50:57.771756 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.771756 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.771766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.771842 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:57.772432 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.772931 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.773622 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.773874 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.773931 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.775137 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:57.775138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:57.775138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:57.775145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:57.776659 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.777041 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.777042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.777042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.777098 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.777274 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.777274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.777275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.777423 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:57.777499 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.777644 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.777716 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:50:57.780280 6164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:57.782047 6164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:57.789079 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.784674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:46591 (local address 127.0.0.1:41268)
0620 14:50:57.784903 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.784946 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:57.785020 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:57.785352 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:57.785354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:57.785356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:57.785431 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.785432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.786537 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.786538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.787711 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.787714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.788675 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.788676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.788782 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.788803 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:50:57.788873 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:50:57.788951 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":52,"threads_started":1}
I20260620 14:50:57.789180 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.784802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.5.118.129:46591)
0620 14:50:57.785168 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.785173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.785182 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:57.785222 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.785224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.785243 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.785350 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:57.785490 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.786437 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.787825 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.788591 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.788666 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.788723 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.789050 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.789101 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":90,"threads_started":1}
I20260620 14:50:57.789397 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.784967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.5.118.130:45611)
0620 14:50:57.785169 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.785174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.785216 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:50:57.785309 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.785311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.785328 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.785404 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:57.785749 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.786911 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.788117 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.788741 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.788815 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.788871 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.789285 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.789330 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260620 14:50:57.789438 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.784998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48534 (local address 127.5.118.131:45531)
0620 14:50:57.785186 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.785190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.785200 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:50:57.785243 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.785245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.785263 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.785359 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:57.785607 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.786910 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.788155 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.788897 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.788958 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.789006 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.789317 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.789358 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260620 14:50:57.789088 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.784704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:45611 (local address 127.0.0.1:55624)
0620 14:50:57.785143 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.785151 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:57.785230 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:57.785522 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:57.785522 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:57.785525 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:57.785563 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.785564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.787045 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.787120 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.788019 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.788021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.788824 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.788824 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.788888 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.788892 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:57.788954 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:50:57.788998 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260620 14:50:57.789208 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.784704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:45531 (local address 127.0.0.1:48534)
0620 14:50:57.784980 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.784988 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:57.785028 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:57.785455 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:57.785455 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:57.785458 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:57.785510 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.785512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.787026 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.787027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.788069 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.788071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.788980 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.788981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.789043 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.789047 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:57.789109 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:50:57.789144 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":160,"threads_started":1}
I20260620 14:50:57.790295 6289 tablet_service.cc:1467] Processing CreateTablet for tablet 435b0d8ebced408386fcc5a8f1e868db (DEFAULT_TABLE table=test-table [id=8f789afaee3b413d926f6a29d29b9b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:57.790441 6557 tablet_service.cc:1467] Processing CreateTablet for tablet 435b0d8ebced408386fcc5a8f1e868db (DEFAULT_TABLE table=test-table [id=8f789afaee3b413d926f6a29d29b9b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:57.790586 6289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435b0d8ebced408386fcc5a8f1e868db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:57.790674 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435b0d8ebced408386fcc5a8f1e868db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:57.792657 6423 tablet_service.cc:1467] Processing CreateTablet for tablet 435b0d8ebced408386fcc5a8f1e868db (DEFAULT_TABLE table=test-table [id=8f789afaee3b413d926f6a29d29b9b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:57.792985 6423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435b0d8ebced408386fcc5a8f1e868db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:57.793234 6640 tablet_bootstrap.cc:492] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1: Bootstrap starting.
I20260620 14:50:57.793828 6640 tablet_bootstrap.cc:654] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:57.794246 6640 log.cc:824] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:57.794549 6641 tablet_bootstrap.cc:492] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2: Bootstrap starting.
I20260620 14:50:57.795084 6640 tablet_bootstrap.cc:492] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1: No bootstrap required, opened a new log
I20260620 14:50:57.795150 6640 ts_tablet_manager.cc:1397] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:50:57.795186 6641 tablet_bootstrap.cc:654] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:57.795747 6641 log.cc:824] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:57.796381 6643 tablet_bootstrap.cc:492] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324: Bootstrap starting.
I20260620 14:50:57.796664 6641 tablet_bootstrap.cc:492] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2: No bootstrap required, opened a new log
I20260620 14:50:57.796733 6641 ts_tablet_manager.cc:1397] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:57.797372 6643 tablet_bootstrap.cc:654] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:57.797447 6640 raft_consensus.cc:348] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a761811e6284230a64a7236312667f1" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 46591 } } peers { permanent_uuid: "83aea54fa49e4120a45872d396721324" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 45611 } } peers { permanent_uuid: "ccbbc41c03594ae8b20e292da6c84ba2" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 45531 } }
I20260620 14:50:57.797636 6640 raft_consensus.cc:374] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:57.797679 6640 raft_consensus.cc:729] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a761811e6284230a64a7236312667f1, State: Initialized, Role: FOLLOWER
I20260620 14:50:57.797824 6640 consensus_queue.cc:260] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a761811e6284230a64a7236312667f1" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 46591 } } peers { permanent_uuid: "83aea54fa49e4120a45872d396721324" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 45611 } } peers { permanent_uuid: "ccbbc41c03594ae8b20e292da6c84ba2" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 45531 } }
I20260620 14:50:57.797869 6643 log.cc:824] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:57.798105 6640 ts_tablet_manager.cc:1428] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:57.798208 6355 heartbeater.cc:503] Master 127.5.118.190:36835 was elected leader, sending a full tablet report...
I20260620 14:50:57.798725 6643 tablet_bootstrap.cc:492] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324: No bootstrap required, opened a new log
I20260620 14:50:57.798791 6643 ts_tablet_manager.cc:1397] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:57.798698 6641 raft_consensus.cc:348] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a761811e6284230a64a7236312667f1" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 46591 } } peers { permanent_uuid: "83aea54fa49e4120a45872d396721324" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 45611 } } peers { permanent_uuid: "ccbbc41c03594ae8b20e292da6c84ba2" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 45531 } }
I20260620 14:50:57.798875 6641 raft_consensus.cc:374] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:57.798921 6641 raft_consensus.cc:729] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbbc41c03594ae8b20e292da6c84ba2, State: Initialized, Role: FOLLOWER
I20260620 14:50:57.799038 6641 consensus_queue.cc:260] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a761811e6284230a64a7236312667f1" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 46591 } } peers { permanent_uuid: "83aea54fa49e4120a45872d396721324" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 45611 } } peers { permanent_uuid: "ccbbc41c03594ae8b20e292da6c84ba2" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 45531 } }
I20260620 14:50:57.799317 6641 ts_tablet_manager.cc:1428] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:57.799330 6623 heartbeater.cc:503] Master 127.5.118.190:36835 was elected leader, sending a full tablet report...
I20260620 14:50:57.800756 6643 raft_consensus.cc:348] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a761811e6284230a64a7236312667f1" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 46591 } } peers { permanent_uuid: "83aea54fa49e4120a45872d396721324" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 45611 } } peers { permanent_uuid: "ccbbc41c03594ae8b20e292da6c84ba2" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 45531 } }
I20260620 14:50:57.800884 6643 raft_consensus.cc:374] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:57.800905 6643 raft_consensus.cc:729] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83aea54fa49e4120a45872d396721324, State: Initialized, Role: FOLLOWER
I20260620 14:50:57.801028 6643 consensus_queue.cc:260] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a761811e6284230a64a7236312667f1" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 46591 } } peers { permanent_uuid: "83aea54fa49e4120a45872d396721324" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 45611 } } peers { permanent_uuid: "ccbbc41c03594ae8b20e292da6c84ba2" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 45531 } }
I20260620 14:50:57.801324 6489 heartbeater.cc:503] Master 127.5.118.190:36835 was elected leader, sending a full tablet report...
I20260620 14:50:57.801334 6643 ts_tablet_manager.cc:1428] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260620 14:50:57.827273 6490 tablet.cc:2369] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:57.892277 6645 raft_consensus.cc:482] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:57.892400 6645 raft_consensus.cc:504] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a761811e6284230a64a7236312667f1" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 46591 } } peers { permanent_uuid: "83aea54fa49e4120a45872d396721324" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 45611 } } peers { permanent_uuid: "ccbbc41c03594ae8b20e292da6c84ba2" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 45531 } }
I20260620 14:50:57.892774 6645 leader_election.cc:288] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83aea54fa49e4120a45872d396721324 (127.5.118.130:45611), ccbbc41c03594ae8b20e292da6c84ba2 (127.5.118.131:45531)
I20260620 14:50:57.895671 6357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.892972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:45531 (local address 127.5.118.129:49431)
0620 14:50:57.893035 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.893047 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:57.893092 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:57.893251 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:57.893252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:57.893253 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:57.893324 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.893326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.893979 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.893980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.894782 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.894785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.895407 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.895408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.895488 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.895491 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:57.895558 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:50:57.895597 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:50:57.895671 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.893060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:49431 (local address 127.5.118.131:45531)
0620 14:50:57.893099 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.893101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.893132 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:50:57.893166 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.893167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.893173 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.893232 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:57.893407 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.893890 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.894873 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.895317 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.895408 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.895417 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.895564 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.895599 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:50:57.895831 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.893000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:45611 (local address 127.5.118.129:42363)
0620 14:50:57.893148 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.893157 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:57.893240 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:57.893390 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:57.893391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:57.893392 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:57.893465 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.893471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.894168 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.894169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.894929 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:57.894931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.895565 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.895566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:57.895643 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.895646 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:50:57.895714 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:50:57.895750 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260620 14:50:57.895825 6638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.893049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:42363 (local address 127.5.118.130:45611)
0620 14:50:57.893115 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.893117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.893224 (+ 107us) server_negotiation.cc:407] Connection header received
0620 14:50:57.893313 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.893314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.893326 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.893385 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:57.893538 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.894078 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.895030 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.895477 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.895555 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.895562 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.895723 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.895757 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:50:57.896085 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b0d8ebced408386fcc5a8f1e868db" candidate_uuid: "3a761811e6284230a64a7236312667f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbbc41c03594ae8b20e292da6c84ba2" is_pre_election: true
I20260620 14:50:57.896122 6443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b0d8ebced408386fcc5a8f1e868db" candidate_uuid: "3a761811e6284230a64a7236312667f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83aea54fa49e4120a45872d396721324" is_pre_election: true
I20260620 14:50:57.896278 6577 raft_consensus.cc:2454] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a761811e6284230a64a7236312667f1 in term 0.
I20260620 14:50:57.896356 6443 raft_consensus.cc:2454] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a761811e6284230a64a7236312667f1 in term 0.
I20260620 14:50:57.896519 6244 leader_election.cc:302] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a761811e6284230a64a7236312667f1, ccbbc41c03594ae8b20e292da6c84ba2; no voters:
I20260620 14:50:57.896668 6645 raft_consensus.cc:2781] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:57.896725 6645 raft_consensus.cc:482] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:57.896749 6645 raft_consensus.cc:3037] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:57.897430 6645 raft_consensus.cc:504] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a761811e6284230a64a7236312667f1" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 46591 } } peers { permanent_uuid: "83aea54fa49e4120a45872d396721324" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 45611 } } peers { permanent_uuid: "ccbbc41c03594ae8b20e292da6c84ba2" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 45531 } }
I20260620 14:50:57.897579 6645 leader_election.cc:288] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [CANDIDATE]: Term 1 election: Requested vote from peers 83aea54fa49e4120a45872d396721324 (127.5.118.130:45611), ccbbc41c03594ae8b20e292da6c84ba2 (127.5.118.131:45531)
I20260620 14:50:57.897718 6443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b0d8ebced408386fcc5a8f1e868db" candidate_uuid: "3a761811e6284230a64a7236312667f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83aea54fa49e4120a45872d396721324"
I20260620 14:50:57.897766 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b0d8ebced408386fcc5a8f1e868db" candidate_uuid: "3a761811e6284230a64a7236312667f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccbbc41c03594ae8b20e292da6c84ba2"
I20260620 14:50:57.897801 6443 raft_consensus.cc:3037] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:57.897840 6577 raft_consensus.cc:3037] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:57.898686 6577 raft_consensus.cc:2454] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a761811e6284230a64a7236312667f1 in term 1.
I20260620 14:50:57.898686 6443 raft_consensus.cc:2454] T 435b0d8ebced408386fcc5a8f1e868db P 83aea54fa49e4120a45872d396721324 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a761811e6284230a64a7236312667f1 in term 1.
I20260620 14:50:57.898901 6243 leader_election.cc:302] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a761811e6284230a64a7236312667f1, 83aea54fa49e4120a45872d396721324; no voters:
I20260620 14:50:57.899019 6645 raft_consensus.cc:2781] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:57.899179 6645 raft_consensus.cc:686] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [term 1 LEADER]: Becoming Leader. State: Replica: 3a761811e6284230a64a7236312667f1, State: Running, Role: LEADER
I20260620 14:50:57.899289 6645 consensus_queue.cc:237] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a761811e6284230a64a7236312667f1" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 46591 } } peers { permanent_uuid: "83aea54fa49e4120a45872d396721324" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 45611 } } peers { permanent_uuid: "ccbbc41c03594ae8b20e292da6c84ba2" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 45531 } }
I20260620 14:50:57.900252 6164 catalog_manager.cc:5495] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a761811e6284230a64a7236312667f1 (127.5.118.129). New cstate: current_term: 1 leader_uuid: "3a761811e6284230a64a7236312667f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a761811e6284230a64a7236312667f1" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 46591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83aea54fa49e4120a45872d396721324" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 45611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccbbc41c03594ae8b20e292da6c84ba2" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 45531 } health_report { overall_health: UNKNOWN } } }
W20260620 14:50:57.908638 6356 tablet.cc:2369] T 435b0d8ebced408386fcc5a8f1e868db P 3a761811e6284230a64a7236312667f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.129@KRBTEST.COM
I20260620 14:50:57.938759 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.930408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.5.118.129:46591)
0620 14:50:57.930483 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.930486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.930554 (+ 68us) server_negotiation.cc:407] Connection header received
0620 14:50:57.930606 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.930607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.930619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.930674 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:57.931358 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.931885 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.932849 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.933061 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.933128 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.935669 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:57.935682 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:57.935682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:57.935689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:57.937373 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.937952 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.937956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.937956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.938024 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.938232 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.938233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.938233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.938390 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:57.938534 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.938616 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.938689 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.118.129@KRBTEST.COM
I20260620 14:50:57.965468 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.957471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.5.118.129:46591)
0620 14:50:57.957544 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.957546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.957650 (+ 104us) server_negotiation.cc:407] Connection header received
0620 14:50:57.957742 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.957743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.957753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.957807 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:57.958411 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.958918 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.959645 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.959865 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.959953 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.962569 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:57.962571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:57.962572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:57.962582 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:57.964072 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.964533 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.964534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.964535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.964600 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.964828 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.964829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.964830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.964944 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:57.965055 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.965251 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.965350 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:50:57 dist-test-slave-2m5f krb5kdc[6119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.129@KRBTEST.COM
I20260620 14:50:57.989055 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.981307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.5.118.129:46591)
0620 14:50:57.981487 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:57.981489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:57.981580 (+ 91us) server_negotiation.cc:407] Connection header received
0620 14:50:57.981696 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:57.981697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:57.981709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:57.981802 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:57.982480 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.983072 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.983910 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:57.984116 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:57.984262 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:57.986587 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:57.986589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:57.986589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:57.986599 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:57.987800 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.988167 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.988209 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.988209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.988283 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:57.988480 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:57.988482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:57.988482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:57.988589 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:57.988705 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:57.988868 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:50:57.988950 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
W20260620 14:50:57.989543 6269 tablet_service.cc:2908] Rejecting scan request for tablet 435b0d8ebced408386fcc5a8f1e868db: Uninitialized: clean time has not yet been initialized
I20260620 14:50:57.990386 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 6226
I20260620 14:50:57.995805 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 6360
W20260620 14:50:58.000345 6624 tablet.cc:2369] T 435b0d8ebced408386fcc5a8f1e868db P ccbbc41c03594ae8b20e292da6c84ba2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:58.000782 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 6494
I20260620 14:50:58.006214 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 6132
2026-06-20T14:50:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2175 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:58 dist-test-slave-2m5f krb5kdc[6664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:58 dist-test-slave-2m5f krb5kdc[6664](info): set up 2 sockets
Jun 20 14:50:58 dist-test-slave-2m5f krb5kdc[6664](info): commencing operation
krb5kdc: starting...
W20260620 14:50:59.140486 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:50:59 dist-test-slave-2m5f krb5kdc[6664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:50:59.263441 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42351
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:42351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:59.344393 6677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:59.344646 6677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:59.344686 6677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:59.345899 6677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:59.345975 6677 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:50:59.345997 6677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:59.346020 6677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:59.346048 6677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:59.346071 6677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:59.347572 6677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:42351
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:59.347849 6677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:59.348074 6677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:59.353111 6683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:59.353098 6682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:59.353184 6685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:59.353202 6677 server_base.cc:1034] running on GCE node
I20260620 14:50:59.353632 6677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:59.353900 6677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:59.355062 6677 hybrid_clock.cc:648] HybridClock initialized: now 1781967059355047 us; error 27 us; skew 500 ppm
Jun 20 14:50:59 dist-test-slave-2m5f krb5kdc[6664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:59.357928 6677 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:50:59.358444 6677 webserver.cc:460] Webserver started at http://127.5.118.190:37961/ using document root <none> and password file <none>
I20260620 14:50:59.358672 6677 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:59.358737 6677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:59.358891 6677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:59.360018 6677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "7fd8ad2718624535be756a4ff71e9f5c"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "aef0316f5f6891d43ed55a9ba74128fe"
server_key_iv: "c5c6ad22fba7fbd54da06577f402f325"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.360417 6677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "7fd8ad2718624535be756a4ff71e9f5c"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "aef0316f5f6891d43ed55a9ba74128fe"
server_key_iv: "c5c6ad22fba7fbd54da06577f402f325"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.361829 6677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:50:59.362602 6692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:59.362798 6677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:50:59.362913 6677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "7fd8ad2718624535be756a4ff71e9f5c"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "aef0316f5f6891d43ed55a9ba74128fe"
server_key_iv: "c5c6ad22fba7fbd54da06577f402f325"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.362995 6677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:59.370549 6677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:59.372813 6677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:59.372973 6677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:59.377861 6677 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:42351
I20260620 14:50:59.377898 6754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:42351 every 8 connection(s)
I20260620 14:50:59.378268 6677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:59.378926 6755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:59.381222 6755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c: Bootstrap starting.
I20260620 14:50:59.381779 6755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:59.382020 6755 log.cc:824] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:59.382689 6755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c: No bootstrap required, opened a new log
I20260620 14:50:59.384886 6755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd8ad2718624535be756a4ff71e9f5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42351 } }
I20260620 14:50:59.385044 6755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:59.385068 6755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd8ad2718624535be756a4ff71e9f5c, State: Initialized, Role: FOLLOWER
I20260620 14:50:59.385177 6755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd8ad2718624535be756a4ff71e9f5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42351 } }
I20260620 14:50:59.385262 6755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:59.385298 6755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:59.385349 6755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:59.385908 6755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd8ad2718624535be756a4ff71e9f5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42351 } }
I20260620 14:50:59.386029 6755 leader_election.cc:302] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fd8ad2718624535be756a4ff71e9f5c; no voters:
I20260620 14:50:59.386222 6755 leader_election.cc:288] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:59.386282 6758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:59.386497 6755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:59.386518 6758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [term 1 LEADER]: Becoming Leader. State: Replica: 7fd8ad2718624535be756a4ff71e9f5c, State: Running, Role: LEADER
I20260620 14:50:59.386638 6758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd8ad2718624535be756a4ff71e9f5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42351 } }
I20260620 14:50:59.387010 6760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fd8ad2718624535be756a4ff71e9f5c. Latest consensus state: current_term: 1 leader_uuid: "7fd8ad2718624535be756a4ff71e9f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd8ad2718624535be756a4ff71e9f5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42351 } } }
I20260620 14:50:59.387104 6760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:59.387256 6759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fd8ad2718624535be756a4ff71e9f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd8ad2718624535be756a4ff71e9f5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42351 } } }
I20260620 14:50:59.387290 6767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:59.387339 6759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:59.387888 6767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:59.388283 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 6677
I20260620 14:50:59.388379 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:50:59.388550 5594 external_mini_cluster.cc:1428] Setting key 84da1b457542bbfe14ff70b18d6b02d4
I20260620 14:50:59.389734 6767 catalog_manager.cc:1347] Generated new cluster ID: 9167ba62e6da4ae49e9ce86e6e9221b9
I20260620 14:50:59.389787 6767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:50:59 dist-test-slave-2m5f krb5kdc[6664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:59.400750 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.389193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.5.118.190:42351)
0620 14:50:59.389371 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:59.389382 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:59.389426 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:50:59.389550 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:59.389557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:59.389614 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:59.389755 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:59.390608 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.392043 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.393250 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.393877 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.393989 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.396937 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:59.396942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:59.396942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:59.396957 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:59.398984 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.399371 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.399375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.399375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.399498 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.399692 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.399692 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.399693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.400047 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:59.400227 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:59.400341 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:50:59.400486 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":108,"threads_started":1}
I20260620 14:50:59.411207 6767 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:59.411949 6767 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:59.414299 6767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c: Generated new TSK 0
I20260620 14:50:59.414551 6767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:50:59.446583 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42351
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:59.546772 6781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:59.547024 6781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:59.547065 6781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:59.548316 6781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:59.548378 6781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:59.548414 6781 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:50:59.548452 6781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:50:59.549942 6781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.118.190:42351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:59.550216 6781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:59.550450 6781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:59.555454 6789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:59.555455 6787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:59.555435 6786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:59.555806 6781 server_base.cc:1034] running on GCE node
I20260620 14:50:59.556070 6781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:59.556360 6781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:59.557530 6781 hybrid_clock.cc:648] HybridClock initialized: now 1781967059557512 us; error 36 us; skew 500 ppm
Jun 20 14:50:59 dist-test-slave-2m5f krb5kdc[6664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:59.560246 6781 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:50:59.560715 6781 webserver.cc:460] Webserver started at http://127.5.118.129:46207/ using document root <none> and password file <none>
I20260620 14:50:59.561048 6781 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:59.561118 6781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:59.561259 6781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:59.562392 6781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "86bc51927bda4216bb1980a10625cc51"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "f577aa2d47f462657419528601c9784d"
server_key_iv: "602a14ed94da4f77f7b7e9fe642481ed"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.562788 6781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "86bc51927bda4216bb1980a10625cc51"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "f577aa2d47f462657419528601c9784d"
server_key_iv: "602a14ed94da4f77f7b7e9fe642481ed"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.564219 6781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:59.565115 6796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:59.565472 6781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:50:59.565584 6781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "86bc51927bda4216bb1980a10625cc51"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "f577aa2d47f462657419528601c9784d"
server_key_iv: "602a14ed94da4f77f7b7e9fe642481ed"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.565668 6781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:59.576342 6781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:59.578464 6781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:59.578630 6781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:59.579185 6781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:59.579244 6781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:59.579288 6781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:59.579319 6781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:59.586256 6781 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:36149
I20260620 14:50:59.586426 6911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:36149 every 8 connection(s)
I20260620 14:50:59.586694 6781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:50:59 dist-test-slave-2m5f krb5kdc[6664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:59.590121 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.579678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:42173 (local address 127.5.118.190:42351)
0620 14:50:59.579881 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:59.579884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:59.580044 (+ 160us) server_negotiation.cc:407] Connection header received
0620 14:50:59.580805 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:59.580807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:59.580823 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:59.580881 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:59.581779 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.582673 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.584355 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.584705 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.584805 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.587519 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:59.587520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:59.587521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:59.587529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:59.588976 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.589344 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.589345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.589346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.589397 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.589602 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.589602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.589602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.589702 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:59.589759 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:59.589990 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:50:59.590056 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260620 14:50:59.590341 6805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.579655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42351 (local address 127.5.118.129:42173)
0620 14:50:59.579925 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.579966 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:59.580702 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:59.580968 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:59.580973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:59.581365 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:59.581659 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.581666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.582795 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.582796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.584233 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.584237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.584889 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.584890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.585018 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.585616 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:59.585627 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:59.587383 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:59.589049 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.589056 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.589057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.589276 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.589453 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.589454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.589454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.589551 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.589754 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:59.589758 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:59.589881 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 14:50:59.589963 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:50:59.590084 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":75,"threads_started":1}
I20260620 14:50:59.590876 6912 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42351
I20260620 14:50:59.590967 6912 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:59.591166 6912 heartbeater.cc:511] Master 127.5.118.190:42351 requested a full tablet report, sending...
I20260620 14:50:59.591722 6708 ts_manager.cc:194] Registered new tserver with Master: 86bc51927bda4216bb1980a10625cc51 (127.5.118.129:36149)
I20260620 14:50:59.592487 6708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:42173
I20260620 14:50:59.593165 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 6781
I20260620 14:50:59.593276 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:59.593443 5594 external_mini_cluster.cc:1428] Setting key df5d80076dde484f5e3378ac2be35267
I20260620 14:50:59.598238 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.591923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:57785 (local address 127.5.118.190:42351)
0620 14:50:59.591964 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:59.591966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:59.592111 (+ 145us) server_negotiation.cc:407] Connection header received
0620 14:50:59.592219 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:59.592220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:59.592230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:59.592288 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:59.592616 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.593273 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.594303 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.594476 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.594564 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.595532 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:59.595533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:59.595534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:59.595540 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:59.597133 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.597472 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.597473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.597474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.597528 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.597722 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.597723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.597723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.597836 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:59.597908 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:59.598054 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:50:59.598134 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:50:59.598234 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.591922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42351 (local address 127.5.118.129:57785)
0620 14:50:59.592048 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.592058 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:59.592127 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:59.592293 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:59.592294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:59.592473 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:59.592522 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.592524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.593386 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.593387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.594122 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.594124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.594563 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.594564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.594640 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.595151 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:59.595155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:59.595432 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:59.597220 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.597221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.597222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.597391 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.597597 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.597598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.597598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.597656 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.597899 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:59.597900 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:59.597947 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:50:59.598029 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:50:59.598134 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:50:59.620679 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42351
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:59.704023 6921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:59.704306 6921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:59.704337 6921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:59.705593 6921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:59.705636 6921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:59.705657 6921 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:50:59.705674 6921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:50:59.707144 6921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.118.190:42351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:59.707386 6921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:59.707638 6921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:50:59.712733 6921 server_base.cc:1034] running on GCE node
W20260620 14:50:59.712708 6929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:59.712715 6926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:59.712834 6927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:59.713092 6921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:59.713340 6921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:59.714481 6921 hybrid_clock.cc:648] HybridClock initialized: now 1781967059714468 us; error 31 us; skew 500 ppm
Jun 20 14:50:59 dist-test-slave-2m5f krb5kdc[6664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:59.717099 6921 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:50:59.717572 6921 webserver.cc:460] Webserver started at http://127.5.118.130:37883/ using document root <none> and password file <none>
I20260620 14:50:59.717796 6921 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:59.717841 6921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:59.717976 6921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:59.719085 6921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "bd14e3cb98a540ddb4b41d383601ad54"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "5f7844e2c330d4d3a100b7f2907e5de8"
server_key_iv: "4b464b9e710c9f99947db89f516133f8"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.719445 6921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "bd14e3cb98a540ddb4b41d383601ad54"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "5f7844e2c330d4d3a100b7f2907e5de8"
server_key_iv: "4b464b9e710c9f99947db89f516133f8"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.720862 6921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:50:59.721649 6936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:59.721828 6921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:50:59.721963 6921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "bd14e3cb98a540ddb4b41d383601ad54"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "5f7844e2c330d4d3a100b7f2907e5de8"
server_key_iv: "4b464b9e710c9f99947db89f516133f8"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.722026 6921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:59.734743 6921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:59.736920 6921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:59.737097 6921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:59.737716 6921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:59.737767 6921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:59.737797 6921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:59.737818 6921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:50:59 dist-test-slave-2m5f krb5kdc[6664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:59.745009 6921 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:33519
I20260620 14:50:59.745188 7051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:33519 every 8 connection(s)
I20260620 14:50:59.745447 6921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:59.745631 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 6921
I20260620 14:50:59.745725 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:59.745913 5594 external_mini_cluster.cc:1428] Setting key 75526ec8e91afef98b2a9dd8ba5477c2
I20260620 14:50:59.747720 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.738081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:38605 (local address 127.5.118.190:42351)
0620 14:50:59.738145 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:59.738147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:59.738416 (+ 269us) server_negotiation.cc:407] Connection header received
0620 14:50:59.738947 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:59.738949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:59.738961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:59.739021 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:59.739739 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.740455 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.741506 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.741723 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.741807 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.744496 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:59.744498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:59.744498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:59.744507 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:59.746248 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.746744 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.746745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.746746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.746811 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.747067 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.747068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.747069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.747190 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:59.747265 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:59.747543 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:50:59.747628 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:50:59.747933 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.738087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42351 (local address 127.5.118.130:38605)
0620 14:50:59.738249 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.738289 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:59.738842 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:59.739096 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:59.739100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:59.739411 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:59.739646 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.739651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.740559 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.740560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.741412 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.741415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.741819 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.741820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.741948 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.742496 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:59.742504 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:59.744397 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:59.746337 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.746345 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.746347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.746665 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.746883 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.746883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.746884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.746999 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.747254 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:59.747260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:59.747411 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 14:50:59.747522 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:50:59.747686 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260620 14:50:59.748407 7052 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42351
I20260620 14:50:59.748524 7052 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:59.748708 7052 heartbeater.cc:511] Master 127.5.118.190:42351 requested a full tablet report, sending...
I20260620 14:50:59.749084 6709 ts_manager.cc:194] Registered new tserver with Master: bd14e3cb98a540ddb4b41d383601ad54 (127.5.118.130:33519)
I20260620 14:50:59.749579 6709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:38605
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
I20260620 14:50:59.755558 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.749638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:42357 (local address 127.5.118.190:42351)
0620 14:50:59.749677 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:59.749680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:59.749803 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:50:59.749873 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:59.749874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:59.749885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:59.749932 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:59.750386 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.750872 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.751599 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.751800 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.751888 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.752878 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:59.752880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:59.752880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:59.752887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:59.754338 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.754698 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.754699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.754700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.754758 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.754989 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.754990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.754990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.755137 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:59.755210 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:59.755383 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:50:59.755462 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:50:59.755564 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.749469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42351 (local address 127.5.118.130:42357)
0620 14:50:59.749739 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.749750 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:59.749818 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:59.749990 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:59.749991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:59.750186 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:59.750274 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.750276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.750967 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.750968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.751502 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.751504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.751889 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.751890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.751970 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.752474 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:59.752479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:59.752767 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:59.754439 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.754440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.754440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.754609 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.754832 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.754833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.754833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.754899 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.755212 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:59.755213 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:59.755259 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:50:59.755354 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:50:59.755464 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":62,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:50:59.773150 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42351
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:37443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:59.856675 7061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:59.856894 7061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:59.856917 7061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:59.858191 7061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:59.858235 7061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:59.858255 7061 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:50:59.858273 7061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:50:59.859653 7061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:37443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.118.190:42351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:50:59.859903 7061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:59.860092 7061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:59.865290 7069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:59.865450 7061 server_base.cc:1034] running on GCE node
W20260620 14:50:59.865324 7066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:59.865288 7067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:59.865844 7061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:59.866101 7061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:59.867265 7061 hybrid_clock.cc:648] HybridClock initialized: now 1781967059867260 us; error 38 us; skew 500 ppm
Jun 20 14:50:59 dist-test-slave-2m5f krb5kdc[6664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:59.869992 7061 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:50:59.870391 7061 webserver.cc:460] Webserver started at http://127.5.118.131:41259/ using document root <none> and password file <none>
I20260620 14:50:59.870604 7061 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:59.870646 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:59.870750 7061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:59.871764 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "20432b1f387d4877b8535b0e6e7aab4c"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "f97a169a3a675a8323d5eb7b441c5328"
server_key_iv: "ef0b51735d21856fb04890821442ab0e"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.872069 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "20432b1f387d4877b8535b0e6e7aab4c"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "f97a169a3a675a8323d5eb7b441c5328"
server_key_iv: "ef0b51735d21856fb04890821442ab0e"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.873400 7061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:50:59.874097 7076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:59.874286 7061 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:50:59.874389 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "20432b1f387d4877b8535b0e6e7aab4c"
format_stamp: "Formatted at 2026-06-20 14:50:59 on dist-test-slave-2m5f"
server_key: "f97a169a3a675a8323d5eb7b441c5328"
server_key_iv: "ef0b51735d21856fb04890821442ab0e"
server_key_version: "encryptionkey@0"
I20260620 14:50:59.874447 7061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:59.892236 7061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:59.894721 7061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:59.894837 7061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:59.895294 7061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:59.895327 7061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:59.895350 7061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:59.895363 7061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:59.902263 7061 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:46463
I20260620 14:50:59.902333 7191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:46463 every 8 connection(s)
I20260620 14:50:59.902686 7061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:50:59 dist-test-slave-2m5f krb5kdc[6664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:59.905880 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.895885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:39665 (local address 127.5.118.190:42351)
0620 14:50:59.895953 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:59.895955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:59.896257 (+ 302us) server_negotiation.cc:407] Connection header received
0620 14:50:59.896869 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:59.896871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:59.896895 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:59.896985 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:59.897910 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.898742 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.900509 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.900697 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.900790 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.903482 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:59.903484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:59.903484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:59.903491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:59.904753 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.905118 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.905119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.905119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.905169 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.905368 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.905368 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.905369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.905472 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:59.905522 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:59.905747 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:50:59.905817 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:50:59.906069 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.895941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42351 (local address 127.5.118.131:39665)
0620 14:50:59.896103 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.896144 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:59.896750 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:59.897068 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:59.897073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:59.897472 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:59.897775 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.897781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.898858 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.898859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.900390 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.900394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.900797 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.900798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.900957 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.901599 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:59.901620 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:59.903395 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:59.904827 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.904833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.904835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.905054 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.905222 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.905223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.905223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.905318 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.905524 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:59.905528 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:59.905642 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 14:50:59.905725 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:50:59.905846 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260620 14:50:59.906534 7192 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42351
I20260620 14:50:59.906615 7192 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:59.906805 7192 heartbeater.cc:511] Master 127.5.118.190:42351 requested a full tablet report, sending...
I20260620 14:50:59.907198 6709 ts_manager.cc:194] Registered new tserver with Master: 20432b1f387d4877b8535b0e6e7aab4c (127.5.118.131:46463)
I20260620 14:50:59.907688 6709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:39665
I20260620 14:50:59.908224 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 7061
I20260620 14:50:59.908314 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:59.908473 5594 external_mini_cluster.cc:1428] Setting key d3503cb0104d70a909ffc1516e367902
I20260620 14:50:59.909242 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:59.913766 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.907670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:33491 (local address 127.5.118.190:42351)
0620 14:50:59.907725 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:59.907727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:59.907862 (+ 135us) server_negotiation.cc:407] Connection header received
0620 14:50:59.907881 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:59.907881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:59.907889 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:59.907928 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:59.908354 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.908986 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.909815 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.910116 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.910193 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.911253 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:59.911255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:59.911255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:59.911261 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:59.912620 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.913029 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.913030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.913030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.913108 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.913337 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.913338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.913338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.913437 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:59.913489 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:59.913648 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:50:59.913706 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:50:59.913892 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.907585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42351 (local address 127.5.118.131:33491)
0620 14:50:59.907734 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.907744 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:59.907802 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:59.907944 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:59.907945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:59.908157 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:59.908265 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.908267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.909089 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.909155 (+ 66us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.909738 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.909740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.910197 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.910198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.910255 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.910711 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:59.910716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:59.911165 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:59.912698 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.912699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.912699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.912947 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.913183 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:59.913184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:59.913185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:59.913262 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:59.913498 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:59.913499 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:59.913551 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 14:50:59.913632 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:50:59.913746 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
Jun 20 14:50:59 dist-test-slave-2m5f krb5kdc[6664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:50:59 dist-test-slave-2m5f krb5kdc[6664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:50:59.932710 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.925735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.5.118.190:42351)
0620 14:50:59.925797 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:59.925799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:59.925811 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:50:59.925876 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:59.925878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:59.925890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:59.925955 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:59.926517 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.927015 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.927758 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.927947 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.928007 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.930265 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:59.930266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:59.930267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:59.930274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:59.931674 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.932017 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.932017 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.932018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.932071 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:59.932297 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:59.932298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:59.932298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:59.932411 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:59.932487 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:59.932578 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:50:59.932641 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:50:59.934504 6709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:59.936100 6709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:59.943089 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.938607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:33519 (local address 127.0.0.1:36630)
0620 14:50:59.939125 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.939133 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:59.939189 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:59.939371 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:59.939371 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:59.939373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:59.939447 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.939449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.940882 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.940883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.941847 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.941849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.942815 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.942816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.942890 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.942895 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:50:59.942926 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 14:50:59.942977 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":9,"thread_start_us":92,"threads_started":1}
I20260620 14:50:59.943152 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.938673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.5.118.129:36149)
0620 14:50:59.939328 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:59.939333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:59.939342 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:50:59.939381 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:59.939383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:59.939399 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:59.939638 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:59.939686 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.940482 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.942052 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.942802 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.942868 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.942931 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:59.942995 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 14:50:59.943036 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":78,"threads_started":1}
I20260620 14:50:59.943174 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.938813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.5.118.131:46463)
0620 14:50:59.938972 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:59.938977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:59.939071 (+ 94us) server_negotiation.cc:407] Connection header received
0620 14:50:59.939120 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:59.939122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:59.939140 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:59.939246 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:59.939541 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.940732 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.942094 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.942625 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.942717 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.942787 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:59.943032 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:50:59.943077 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260620 14:50:59.943092 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.938577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:36149 (local address 127.0.0.1:54436)
0620 14:50:59.938772 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.938823 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:59.938889 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:59.939515 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:59.939516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:59.939518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:59.939568 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.939569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.940569 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.940570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.941961 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.941964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.942867 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.942868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.942923 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.942927 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:50:59.942973 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 14:50:59.943013 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":32,"thread_start_us":72,"threads_started":1}
I20260620 14:50:59.943608 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.938607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:46463 (local address 127.0.0.1:50050)
0620 14:50:59.939000 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.939010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:59.939061 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:59.939347 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:59.939351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:59.939354 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:59.939450 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.939452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.940838 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.940839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.941982 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:59.941983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.942716 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.942717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:59.942800 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.942816 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:50:59.942890 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:50:59.942949 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":76,"thread_start_us":92,"threads_started":1}
I20260620 14:50:59.943953 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.938944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.5.118.130:33519)
0620 14:50:59.939117 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:59.939121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:59.939173 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:50:59.939249 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:59.939251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:59.939264 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:59.939526 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:59.939587 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.940775 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.941956 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:59.942706 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:59.943727 (+ 1021us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:59.943798 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:59.943825 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:50:59.943866 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":92,"threads_started":1}
I20260620 14:50:59.944216 6846 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc616a4642c443595171bee5366fb97 (DEFAULT_TABLE table=test-table [id=28e7f593dcd8446bb026aa1b2c9b3aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:59.944391 7126 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc616a4642c443595171bee5366fb97 (DEFAULT_TABLE table=test-table [id=28e7f593dcd8446bb026aa1b2c9b3aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:59.944615 6846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc616a4642c443595171bee5366fb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:59.944731 7126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc616a4642c443595171bee5366fb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:59.944717 6986 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc616a4642c443595171bee5366fb97 (DEFAULT_TABLE table=test-table [id=28e7f593dcd8446bb026aa1b2c9b3aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:59.945031 6986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc616a4642c443595171bee5366fb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:59.947541 7215 tablet_bootstrap.cc:492] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51: Bootstrap starting.
I20260620 14:50:59.948107 7216 tablet_bootstrap.cc:492] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c: Bootstrap starting.
I20260620 14:50:59.948307 7215 tablet_bootstrap.cc:654] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:59.948737 7215 log.cc:824] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:59.948823 7217 tablet_bootstrap.cc:492] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: Bootstrap starting.
I20260620 14:50:59.949229 7216 tablet_bootstrap.cc:654] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:59.949460 7217 tablet_bootstrap.cc:654] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:59.949553 7215 tablet_bootstrap.cc:492] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51: No bootstrap required, opened a new log
I20260620 14:50:59.949630 7215 ts_tablet_manager.cc:1397] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:50:59.949626 7216 log.cc:824] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:59.949849 7217 log.cc:824] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:59.950356 7216 tablet_bootstrap.cc:492] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c: No bootstrap required, opened a new log
I20260620 14:50:59.950412 7216 ts_tablet_manager.cc:1397] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:59.951413 7215 raft_consensus.cc:348] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } }
I20260620 14:50:59.951578 7215 raft_consensus.cc:374] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:59.951625 7215 raft_consensus.cc:729] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc51927bda4216bb1980a10625cc51, State: Initialized, Role: FOLLOWER
I20260620 14:50:59.951751 7215 consensus_queue.cc:260] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } }
I20260620 14:50:59.952064 7215 ts_tablet_manager.cc:1428] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:50:59.952215 6912 heartbeater.cc:503] Master 127.5.118.190:42351 was elected leader, sending a full tablet report...
I20260620 14:50:59.952253 7216 raft_consensus.cc:348] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } }
I20260620 14:50:59.952411 7216 raft_consensus.cc:374] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:59.952453 7216 raft_consensus.cc:729] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20432b1f387d4877b8535b0e6e7aab4c, State: Initialized, Role: FOLLOWER
I20260620 14:50:59.952567 7216 consensus_queue.cc:260] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } }
I20260620 14:50:59.952858 7216 ts_tablet_manager.cc:1428] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:50:59.952877 7192 heartbeater.cc:503] Master 127.5.118.190:42351 was elected leader, sending a full tablet report...
I20260620 14:50:59.953547 7217 tablet_bootstrap.cc:492] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: No bootstrap required, opened a new log
I20260620 14:50:59.953625 7217 ts_tablet_manager.cc:1397] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 14:50:59.955148 7217 raft_consensus.cc:348] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } }
I20260620 14:50:59.955318 7217 raft_consensus.cc:374] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:59.955367 7217 raft_consensus.cc:729] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd14e3cb98a540ddb4b41d383601ad54, State: Initialized, Role: FOLLOWER
I20260620 14:50:59.955489 7217 consensus_queue.cc:260] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } }
I20260620 14:50:59.955760 7217 ts_tablet_manager.cc:1428] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:50:59.955828 7052 heartbeater.cc:503] Master 127.5.118.190:42351 was elected leader, sending a full tablet report...
W20260620 14:50:59.996165 7053 tablet.cc:2369] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:00.016412 7222 raft_consensus.cc:482] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:00.016548 7222 raft_consensus.cc:504] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } }
I20260620 14:51:00.016923 7222 leader_election.cc:288] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86bc51927bda4216bb1980a10625cc51 (127.5.118.129:36149), bd14e3cb98a540ddb4b41d383601ad54 (127.5.118.130:33519)
I20260620 14:51:00.020089 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.017233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:52713 (local address 127.5.118.130:33519)
0620 14:51:00.017332 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:00.017334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:00.017441 (+ 107us) server_negotiation.cc:407] Connection header received
0620 14:51:00.017496 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:00.017497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:00.017508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:00.017568 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:00.017819 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.018288 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.019320 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.019683 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.019868 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.019881 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:00.019983 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 14:51:00.020030 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:51:00.020085 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.017247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:41375 (local address 127.5.118.129:36149)
0620 14:51:00.017296 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:00.017299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:00.017322 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:51:00.017356 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:00.017357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:00.017370 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:00.017448 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:00.017619 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.018226 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.019224 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.019669 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.019733 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.019741 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:00.019951 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:51:00.020002 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:51:00.020087 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.017140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:36149 (local address 127.5.118.131:41375)
0620 14:51:00.017202 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.017226 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:00.017272 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:00.017460 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:00.017461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:00.017462 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:00.017544 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.017546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.018318 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.018319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.019123 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.019125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.019769 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.019770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.019862 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.019867 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:51:00.019927 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:51:00.019997 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":25}
I20260620 14:51:00.020103 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.017140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:33519 (local address 127.5.118.131:52713)
0620 14:51:00.017338 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.017349 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:00.017438 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:00.017601 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:00.017603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:00.017605 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:00.017691 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.017692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.018379 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.018380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.019213 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.019215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.019820 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.019821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.019885 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.019888 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:00.019969 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:51:00.020012 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":13,"thread_start_us":110,"threads_started":1}
I20260620 14:51:00.020503 6866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc616a4642c443595171bee5366fb97" candidate_uuid: "20432b1f387d4877b8535b0e6e7aab4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc51927bda4216bb1980a10625cc51" is_pre_election: true
I20260620 14:51:00.020704 6866 raft_consensus.cc:2454] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20432b1f387d4877b8535b0e6e7aab4c in term 0.
I20260620 14:51:00.020709 7006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc616a4642c443595171bee5366fb97" candidate_uuid: "20432b1f387d4877b8535b0e6e7aab4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd14e3cb98a540ddb4b41d383601ad54" is_pre_election: true
I20260620 14:51:00.020843 7006 raft_consensus.cc:2454] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20432b1f387d4877b8535b0e6e7aab4c in term 0.
I20260620 14:51:00.021099 7080 leader_election.cc:302] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20432b1f387d4877b8535b0e6e7aab4c, bd14e3cb98a540ddb4b41d383601ad54; no voters:
I20260620 14:51:00.021288 7222 raft_consensus.cc:2781] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:00.021360 7222 raft_consensus.cc:482] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:00.021380 7222 raft_consensus.cc:3037] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:00.022099 7222 raft_consensus.cc:504] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } }
I20260620 14:51:00.022253 7222 leader_election.cc:288] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [CANDIDATE]: Term 1 election: Requested vote from peers 86bc51927bda4216bb1980a10625cc51 (127.5.118.129:36149), bd14e3cb98a540ddb4b41d383601ad54 (127.5.118.130:33519)
I20260620 14:51:00.022437 6866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc616a4642c443595171bee5366fb97" candidate_uuid: "20432b1f387d4877b8535b0e6e7aab4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc51927bda4216bb1980a10625cc51"
I20260620 14:51:00.022444 7006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fc616a4642c443595171bee5366fb97" candidate_uuid: "20432b1f387d4877b8535b0e6e7aab4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd14e3cb98a540ddb4b41d383601ad54"
I20260620 14:51:00.022528 6866 raft_consensus.cc:3037] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:00.022528 7006 raft_consensus.cc:3037] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:00.023487 7006 raft_consensus.cc:2454] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20432b1f387d4877b8535b0e6e7aab4c in term 1.
I20260620 14:51:00.023488 6866 raft_consensus.cc:2454] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20432b1f387d4877b8535b0e6e7aab4c in term 1.
I20260620 14:51:00.024060 7077 leader_election.cc:302] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20432b1f387d4877b8535b0e6e7aab4c, 86bc51927bda4216bb1980a10625cc51; no voters:
I20260620 14:51:00.024248 7222 raft_consensus.cc:2781] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:00.024468 7222 raft_consensus.cc:686] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 1 LEADER]: Becoming Leader. State: Replica: 20432b1f387d4877b8535b0e6e7aab4c, State: Running, Role: LEADER
I20260620 14:51:00.024591 7222 consensus_queue.cc:237] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } }
I20260620 14:51:00.025631 6709 catalog_manager.cc:5495] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 20432b1f387d4877b8535b0e6e7aab4c (127.5.118.131). New cstate: current_term: 1 leader_uuid: "20432b1f387d4877b8535b0e6e7aab4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:00.037890 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.035210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.5.118.131:46463)
0620 14:51:00.035276 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:00.035278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:00.035287 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:00.035408 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:00.035409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:00.035429 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:00.035501 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:00.035664 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.036141 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.037019 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.037212 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.037299 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.037515 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:00.037615 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:00.037696 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:00.037749 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 14:51:00.037775 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:51:00.039880 6866 raft_consensus.cc:1261] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51 [term 1 FOLLOWER]: Refusing update from remote peer 20432b1f387d4877b8535b0e6e7aab4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:00.039888 7006 raft_consensus.cc:1261] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54 [term 1 FOLLOWER]: Refusing update from remote peer 20432b1f387d4877b8535b0e6e7aab4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:00.040318 7225 consensus_queue.cc:1035] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:00.040445 7222 consensus_queue.cc:1035] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:00.042665 7228 mvcc.cc:205] Tried to move back new op lower bound from 7298937077920120832 to 7298937077861527552. Current Snapshot: MvccSnapshot[applied={T|T < 7298937077920120832}]
I20260620 14:51:00.046691 6709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:00.046872 6709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:00.047781 6709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c: Sending DeleteTablet for 3 replicas of tablet 8fc616a4642c443595171bee5366fb97
I20260620 14:51:00.048117 7126 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc616a4642c443595171bee5366fb97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:00 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:50050
I20260620 14:51:00.048192 6986 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc616a4642c443595171bee5366fb97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:00 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:36630
I20260620 14:51:00.048216 6846 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc616a4642c443595171bee5366fb97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:00 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:54436
I20260620 14:51:00.048453 7238 tablet_replica.cc:330] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: stopping tablet replica
I20260620 14:51:00.048571 7238 raft_consensus.cc:2229] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:00.048707 7238 raft_consensus.cc:2258] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:00.048799 6709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:51:00.048895 7239 tablet_replica.cc:330] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51: stopping tablet replica
W20260620 14:51:00.048945 6709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:00.049006 7239 raft_consensus.cc:2229] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:00.049252 7238 ts_tablet_manager.cc:1905] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:00.049461 7239 raft_consensus.cc:2258] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:00.050478 7126 tablet_service.cc:1467] Processing CreateTablet for tablet b505c4ff20eb4b61b0e2ab6f4dc02508 (DEFAULT_TABLE table=test-table [id=3c047e095a29468690a188fafa686a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:00.050485 6846 tablet_service.cc:1467] Processing CreateTablet for tablet b505c4ff20eb4b61b0e2ab6f4dc02508 (DEFAULT_TABLE table=test-table [id=3c047e095a29468690a188fafa686a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:00.050606 7126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b505c4ff20eb4b61b0e2ab6f4dc02508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:00.050606 6846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b505c4ff20eb4b61b0e2ab6f4dc02508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:00.051968 7216 tablet_bootstrap.cc:492] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c: Bootstrap starting.
I20260620 14:51:00.052465 7216 tablet_bootstrap.cc:654] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:00.052590 7215 tablet_bootstrap.cc:492] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51: Bootstrap starting.
I20260620 14:51:00.053025 7239 ts_tablet_manager.cc:1905] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:00.053025 6986 tablet_service.cc:1467] Processing CreateTablet for tablet b505c4ff20eb4b61b0e2ab6f4dc02508 (DEFAULT_TABLE table=test-table [id=3c047e095a29468690a188fafa686a6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:00.052598 7237 tablet_replica.cc:330] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c: stopping tablet replica
I20260620 14:51:00.053144 6986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b505c4ff20eb4b61b0e2ab6f4dc02508. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:00.053268 7237 raft_consensus.cc:2229] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:00.053444 7237 raft_consensus.cc:2258] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:00.053365 7215 tablet_bootstrap.cc:654] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:00.054522 7238 ts_tablet_manager.cc:1918] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:00.054572 7217 tablet_bootstrap.cc:492] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54: Bootstrap starting.
I20260620 14:51:00.054613 7238 log.cc:1196] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/8fc616a4642c443595171bee5366fb97
I20260620 14:51:00.054920 7238 ts_tablet_manager.cc:1939] T 8fc616a4642c443595171bee5366fb97 P bd14e3cb98a540ddb4b41d383601ad54: Deleting consensus metadata
I20260620 14:51:00.055009 7217 tablet_bootstrap.cc:654] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:00.055433 7239 ts_tablet_manager.cc:1918] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:00.055455 7237 ts_tablet_manager.cc:1905] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:00.055568 7239 log.cc:1196] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/wal/wals/8fc616a4642c443595171bee5366fb97
I20260620 14:51:00.055876 7239 ts_tablet_manager.cc:1939] T 8fc616a4642c443595171bee5366fb97 P 86bc51927bda4216bb1980a10625cc51: Deleting consensus metadata
I20260620 14:51:00.055868 6696 catalog_manager.cc:4841] TS bd14e3cb98a540ddb4b41d383601ad54 (127.5.118.130:33519): tablet 8fc616a4642c443595171bee5366fb97 (table test-table [id=28e7f593dcd8446bb026aa1b2c9b3aa1]) successfully deleted
I20260620 14:51:00.056329 7217 tablet_bootstrap.cc:492] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54: No bootstrap required, opened a new log
I20260620 14:51:00.056357 6693 catalog_manager.cc:4841] TS 86bc51927bda4216bb1980a10625cc51 (127.5.118.129:36149): tablet 8fc616a4642c443595171bee5366fb97 (table test-table [id=28e7f593dcd8446bb026aa1b2c9b3aa1]) successfully deleted
I20260620 14:51:00.056384 7217 ts_tablet_manager.cc:1397] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:00.056547 7217 raft_consensus.cc:348] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.056644 7217 raft_consensus.cc:374] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:00.056679 7217 raft_consensus.cc:729] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd14e3cb98a540ddb4b41d383601ad54, State: Initialized, Role: FOLLOWER
I20260620 14:51:00.056730 7217 consensus_queue.cc:260] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.056849 7217 ts_tablet_manager.cc:1428] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:00.057211 7237 ts_tablet_manager.cc:1918] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:00.057304 7237 log.cc:1196] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/8fc616a4642c443595171bee5366fb97
I20260620 14:51:00.057588 7237 ts_tablet_manager.cc:1939] T 8fc616a4642c443595171bee5366fb97 P 20432b1f387d4877b8535b0e6e7aab4c: Deleting consensus metadata
I20260620 14:51:00.057821 7215 tablet_bootstrap.cc:492] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51: No bootstrap required, opened a new log
I20260620 14:51:00.057875 7215 ts_tablet_manager.cc:1397] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260620 14:51:00.058017 6695 catalog_manager.cc:4841] TS 20432b1f387d4877b8535b0e6e7aab4c (127.5.118.131:46463): tablet 8fc616a4642c443595171bee5366fb97 (table test-table [id=28e7f593dcd8446bb026aa1b2c9b3aa1]) successfully deleted
I20260620 14:51:00.058038 7215 raft_consensus.cc:348] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.058141 7215 raft_consensus.cc:374] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:00.058177 7215 raft_consensus.cc:729] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc51927bda4216bb1980a10625cc51, State: Initialized, Role: FOLLOWER
I20260620 14:51:00.058243 7215 consensus_queue.cc:260] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.058272 7216 tablet_bootstrap.cc:492] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c: No bootstrap required, opened a new log
I20260620 14:51:00.058321 7216 ts_tablet_manager.cc:1397] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 14:51:00.058353 7215 ts_tablet_manager.cc:1428] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:00.058485 7216 raft_consensus.cc:348] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.058576 7216 raft_consensus.cc:374] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:00.058602 7216 raft_consensus.cc:729] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20432b1f387d4877b8535b0e6e7aab4c, State: Initialized, Role: FOLLOWER
I20260620 14:51:00.058652 7216 consensus_queue.cc:260] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.058750 7216 ts_tablet_manager.cc:1428] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20260620 14:51:00.087450 6913 tablet.cc:2369] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:00.112850 7221 raft_consensus.cc:482] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:00.112950 7221 raft_consensus.cc:504] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.113310 7221 leader_election.cc:288] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd14e3cb98a540ddb4b41d383601ad54 (127.5.118.130:33519), 20432b1f387d4877b8535b0e6e7aab4c (127.5.118.131:46463)
I20260620 14:51:00.116843 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.113670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:46493 (local address 127.5.118.131:46463)
0620 14:51:00.113730 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:00.113732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:00.113742 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:00.113769 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:00.113770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:00.113782 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:00.113845 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:00.114024 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.114735 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.116017 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.116464 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.116556 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.116567 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:00.116724 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:51:00.116768 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:51:00.116875 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.113483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:46463 (local address 127.5.118.129:46493)
0620 14:51:00.113613 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.113624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:00.113680 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:00.113857 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:00.113858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:00.113861 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:00.113942 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.113944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.114821 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.114822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.115889 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.115892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.116561 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.116562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.116656 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.116661 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:51:00.116717 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:51:00.116762 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":4,"thread_start_us":59,"threads_started":1}
I20260620 14:51:00.116997 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.113577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:53539 (local address 127.5.118.130:33519)
0620 14:51:00.113624 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:00.113626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:00.113790 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:51:00.113866 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:00.113867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:00.113876 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:00.113906 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:00.114077 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.114543 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.116002 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.116578 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.116657 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.116666 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:00.116860 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:51:00.116914 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:51:00.117218 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.113485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:33519 (local address 127.5.118.129:53539)
0620 14:51:00.113724 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.113734 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:00.113798 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:00.113968 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:00.113969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:00.113971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:00.114025 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.114026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.114639 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.114640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.115889 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.115892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.116673 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.116674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.116759 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.116762 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:00.117073 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 14:51:00.117119 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260620 14:51:00.117446 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b505c4ff20eb4b61b0e2ab6f4dc02508" candidate_uuid: "86bc51927bda4216bb1980a10625cc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20432b1f387d4877b8535b0e6e7aab4c" is_pre_election: true
I20260620 14:51:00.117578 7146 raft_consensus.cc:2454] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc51927bda4216bb1980a10625cc51 in term 0.
I20260620 14:51:00.117986 6799 leader_election.cc:302] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20432b1f387d4877b8535b0e6e7aab4c, 86bc51927bda4216bb1980a10625cc51; no voters:
I20260620 14:51:00.118142 7221 raft_consensus.cc:2781] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:00.118186 7221 raft_consensus.cc:482] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:00.118206 7221 raft_consensus.cc:3037] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:00.118921 7221 raft_consensus.cc:504] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.119079 7221 leader_election.cc:288] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [CANDIDATE]: Term 1 election: Requested vote from peers bd14e3cb98a540ddb4b41d383601ad54 (127.5.118.130:33519), 20432b1f387d4877b8535b0e6e7aab4c (127.5.118.131:46463)
I20260620 14:51:00.117825 7006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b505c4ff20eb4b61b0e2ab6f4dc02508" candidate_uuid: "86bc51927bda4216bb1980a10625cc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd14e3cb98a540ddb4b41d383601ad54" is_pre_election: true
I20260620 14:51:00.119199 7006 raft_consensus.cc:2454] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc51927bda4216bb1980a10625cc51 in term 0.
I20260620 14:51:00.119243 7005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b505c4ff20eb4b61b0e2ab6f4dc02508" candidate_uuid: "86bc51927bda4216bb1980a10625cc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd14e3cb98a540ddb4b41d383601ad54"
I20260620 14:51:00.119320 7005 raft_consensus.cc:3037] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:00.119452 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b505c4ff20eb4b61b0e2ab6f4dc02508" candidate_uuid: "86bc51927bda4216bb1980a10625cc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20432b1f387d4877b8535b0e6e7aab4c"
I20260620 14:51:00.119529 7146 raft_consensus.cc:3037] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:00.120163 7005 raft_consensus.cc:2454] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc51927bda4216bb1980a10625cc51 in term 1.
I20260620 14:51:00.120411 6800 leader_election.cc:302] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc51927bda4216bb1980a10625cc51, bd14e3cb98a540ddb4b41d383601ad54; no voters:
I20260620 14:51:00.120445 7146 raft_consensus.cc:2454] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc51927bda4216bb1980a10625cc51 in term 1.
I20260620 14:51:00.120508 7221 raft_consensus.cc:2781] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:00.120625 7221 raft_consensus.cc:686] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 1 LEADER]: Becoming Leader. State: Replica: 86bc51927bda4216bb1980a10625cc51, State: Running, Role: LEADER
I20260620 14:51:00.120698 7221 consensus_queue.cc:237] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.121474 6708 catalog_manager.cc:5495] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc51927bda4216bb1980a10625cc51 (127.5.118.129). New cstate: current_term: 1 leader_uuid: "86bc51927bda4216bb1980a10625cc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } health_report { overall_health: UNKNOWN } } }
W20260620 14:51:00.153246 7193 tablet.cc:2369] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:51:00 dist-test-slave-2m5f krb5kdc[6664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:00.161903 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.148997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42351 (local address 127.0.0.1:52342)
0620 14:51:00.149133 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.149145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:00.149217 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:00.149482 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:00.149484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:00.149859 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:00.149935 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.149938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.150934 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.150935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.151467 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.151469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.151933 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.151934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.151985 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.157075 (+ 5090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:00.157081 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:00.158559 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:00.160662 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:00.160671 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:00.160673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:00.160922 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:00.161154 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:00.161155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:00.161155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:00.161230 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:00.161543 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:00.161551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:00.161611 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 14:51:00.161685 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:51:00.161814 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260620 14:51:00.162003 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.149036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.5.118.190:42351)
0620 14:51:00.149114 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:00.149117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:00.149216 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:51:00.149298 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:00.149307 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:00.149324 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:00.149411 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:00.150063 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.150820 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.151560 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.151830 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.151917 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.158671 (+ 6754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:00.158672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:00.158673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:00.158682 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:00.160559 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:00.161003 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:00.161004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:00.161005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:00.161069 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:00.161305 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:00.161306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:00.161306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:00.161451 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:00.161520 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:00.161727 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 14:51:00.161886 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":39}
I20260620 14:51:00.164392 6708 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:52342:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:51:00.164613 6708 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:00.166128 6846 tablet_service.cc:1467] Processing CreateTablet for tablet fb80a922687d4aa98c794c22b20d4535 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c2fce79642a4ad9b4585bfe9424ae4c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:51:00.166288 6846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb80a922687d4aa98c794c22b20d4535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:00.166247 7126 tablet_service.cc:1467] Processing CreateTablet for tablet fb80a922687d4aa98c794c22b20d4535 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c2fce79642a4ad9b4585bfe9424ae4c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:51:00.166241 6986 tablet_service.cc:1467] Processing CreateTablet for tablet fb80a922687d4aa98c794c22b20d4535 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c2fce79642a4ad9b4585bfe9424ae4c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:51:00.166394 7126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb80a922687d4aa98c794c22b20d4535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:00.166405 6986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb80a922687d4aa98c794c22b20d4535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:00.167989 7217 tablet_bootstrap.cc:492] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54: Bootstrap starting.
I20260620 14:51:00.168267 7216 tablet_bootstrap.cc:492] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c: Bootstrap starting.
I20260620 14:51:00.168433 7215 tablet_bootstrap.cc:492] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: Bootstrap starting.
I20260620 14:51:00.168624 7216 tablet_bootstrap.cc:654] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:00.168839 7215 tablet_bootstrap.cc:654] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:00.169384 7217 tablet_bootstrap.cc:654] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:00.169713 7215 tablet_bootstrap.cc:492] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: No bootstrap required, opened a new log
I20260620 14:51:00.169761 7215 ts_tablet_manager.cc:1397] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:00.169904 7215 raft_consensus.cc:348] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.169983 7215 raft_consensus.cc:374] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:00.170007 7215 raft_consensus.cc:729] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc51927bda4216bb1980a10625cc51, State: Initialized, Role: FOLLOWER
I20260620 14:51:00.170032 7217 tablet_bootstrap.cc:492] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54: No bootstrap required, opened a new log
I20260620 14:51:00.170078 7217 ts_tablet_manager.cc:1397] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:51:00.170053 7215 consensus_queue.cc:260] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.170154 7215 ts_tablet_manager.cc:1428] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:00.170226 7217 raft_consensus.cc:348] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.170291 7217 raft_consensus.cc:374] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:00.170305 7216 tablet_bootstrap.cc:492] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c: No bootstrap required, opened a new log
I20260620 14:51:00.170321 7217 raft_consensus.cc:729] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd14e3cb98a540ddb4b41d383601ad54, State: Initialized, Role: FOLLOWER
I20260620 14:51:00.170347 7216 ts_tablet_manager.cc:1397] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:51:00.170306 7221 tablet_replica.cc:439] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } }
I20260620 14:51:00.170377 7221 tablet_replica.cc:442] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:00.170368 7217 consensus_queue.cc:260] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.170447 7217 ts_tablet_manager.cc:1428] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:00.170447 7216 raft_consensus.cc:348] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.170506 7216 raft_consensus.cc:374] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:00.170527 7216 raft_consensus.cc:729] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20432b1f387d4877b8535b0e6e7aab4c, State: Initialized, Role: FOLLOWER
I20260620 14:51:00.170581 7223 tablet_replica.cc:439] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } }
I20260620 14:51:00.170583 7216 consensus_queue.cc:260] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.170635 7223 tablet_replica.cc:442] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:00.170677 7216 ts_tablet_manager.cc:1428] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:00.170799 7222 tablet_replica.cc:439] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } }
I20260620 14:51:00.170872 7222 tablet_replica.cc:442] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:00.432991 7221 raft_consensus.cc:482] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:00.433102 7221 raft_consensus.cc:504] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.433302 7221 leader_election.cc:288] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd14e3cb98a540ddb4b41d383601ad54 (127.5.118.130:33519), 20432b1f387d4877b8535b0e6e7aab4c (127.5.118.131:46463)
I20260620 14:51:00.433575 7005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb80a922687d4aa98c794c22b20d4535" candidate_uuid: "86bc51927bda4216bb1980a10625cc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd14e3cb98a540ddb4b41d383601ad54" is_pre_election: true
I20260620 14:51:00.433565 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb80a922687d4aa98c794c22b20d4535" candidate_uuid: "86bc51927bda4216bb1980a10625cc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20432b1f387d4877b8535b0e6e7aab4c" is_pre_election: true
I20260620 14:51:00.433665 7005 raft_consensus.cc:2454] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc51927bda4216bb1980a10625cc51 in term 0.
I20260620 14:51:00.433691 7146 raft_consensus.cc:2454] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc51927bda4216bb1980a10625cc51 in term 0.
I20260620 14:51:00.433851 6800 leader_election.cc:302] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc51927bda4216bb1980a10625cc51, bd14e3cb98a540ddb4b41d383601ad54; no voters:
I20260620 14:51:00.433954 7221 raft_consensus.cc:2781] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:00.433984 7221 raft_consensus.cc:482] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:00.434006 7221 raft_consensus.cc:3037] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:00.434671 7221 raft_consensus.cc:504] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.434819 7221 leader_election.cc:288] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [CANDIDATE]: Term 1 election: Requested vote from peers bd14e3cb98a540ddb4b41d383601ad54 (127.5.118.130:33519), 20432b1f387d4877b8535b0e6e7aab4c (127.5.118.131:46463)
I20260620 14:51:00.434964 7005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb80a922687d4aa98c794c22b20d4535" candidate_uuid: "86bc51927bda4216bb1980a10625cc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd14e3cb98a540ddb4b41d383601ad54"
I20260620 14:51:00.435046 7005 raft_consensus.cc:3037] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:00.435068 7146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb80a922687d4aa98c794c22b20d4535" candidate_uuid: "86bc51927bda4216bb1980a10625cc51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20432b1f387d4877b8535b0e6e7aab4c"
I20260620 14:51:00.435155 7146 raft_consensus.cc:3037] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:00.435667 7005 raft_consensus.cc:2454] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc51927bda4216bb1980a10625cc51 in term 1.
I20260620 14:51:00.435842 6800 leader_election.cc:302] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86bc51927bda4216bb1980a10625cc51, bd14e3cb98a540ddb4b41d383601ad54; no voters:
I20260620 14:51:00.435858 7146 raft_consensus.cc:2454] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc51927bda4216bb1980a10625cc51 in term 1.
I20260620 14:51:00.435966 7221 raft_consensus.cc:2781] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:00.436031 7221 raft_consensus.cc:686] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [term 1 LEADER]: Becoming Leader. State: Replica: 86bc51927bda4216bb1980a10625cc51, State: Running, Role: LEADER
I20260620 14:51:00.436086 7221 consensus_queue.cc:237] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } }
I20260620 14:51:00.436467 7245 tablet_replica.cc:439] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: TxnStatusTablet state changed. Reason: New leader 86bc51927bda4216bb1980a10625cc51. Latest consensus state: current_term: 1 leader_uuid: "86bc51927bda4216bb1980a10625cc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } }
I20260620 14:51:00.436575 7245 tablet_replica.cc:442] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:51:00.436755 7257 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:51:00.436902 7257 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:51:00.436928 6709 catalog_manager.cc:5495] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc51927bda4216bb1980a10625cc51 (127.5.118.129). New cstate: current_term: 1 leader_uuid: "86bc51927bda4216bb1980a10625cc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:00.458630 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.455980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:36149 (local address 127.0.0.1:54444)
0620 14:51:00.456066 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.456091 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:00.456145 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:00.456390 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:00.456391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:00.456392 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:51:00.456450 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.456452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.457069 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.457070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.457680 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.457683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.458062 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.458063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.458132 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.458155 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:51:00.458439 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:51:00.458449 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:51:00.458504 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:51:00.458553 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:51:00.458606 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.456055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.5.118.129:36149)
0620 14:51:00.456122 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:00.456124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:00.456133 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:00.456295 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:00.456296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:00.456314 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:00.456387 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:00.456516 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.456988 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.457786 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.457978 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.458061 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.458260 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:00.458373 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:00.458456 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:00.458499 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:51:00.458522 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:51:00.459939 7005 raft_consensus.cc:1261] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54 [term 1 FOLLOWER]: Refusing update from remote peer 86bc51927bda4216bb1980a10625cc51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:00.460005 7146 raft_consensus.cc:1261] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c [term 1 FOLLOWER]: Refusing update from remote peer 86bc51927bda4216bb1980a10625cc51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:00.460256 7221 consensus_queue.cc:1035] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:00.460373 7245 consensus_queue.cc:1035] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:00.461009 7222 tablet_replica.cc:439] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c: TxnStatusTablet state changed. Reason: New leader 86bc51927bda4216bb1980a10625cc51. Latest consensus state: current_term: 1 leader_uuid: "86bc51927bda4216bb1980a10625cc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } }
I20260620 14:51:00.461109 7222 tablet_replica.cc:442] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:00.461092 7223 tablet_replica.cc:439] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54: TxnStatusTablet state changed. Reason: New leader 86bc51927bda4216bb1980a10625cc51. Latest consensus state: current_term: 1 leader_uuid: "86bc51927bda4216bb1980a10625cc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } }
I20260620 14:51:00.461185 7223 tablet_replica.cc:442] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:00.461818 7221 tablet_replica.cc:439] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86bc51927bda4216bb1980a10625cc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } }
I20260620 14:51:00.461907 7221 tablet_replica.cc:442] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:51:00.461969 7230 mvcc.cc:205] Tried to move back new op lower bound from 7298937079641628672 to 7298937079546679296. Current Snapshot: MvccSnapshot[applied={T|T < 7298937079641628672 or (T in {7298937079641628672})}]
I20260620 14:51:00.462119 7222 tablet_replica.cc:439] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86bc51927bda4216bb1980a10625cc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } }
I20260620 14:51:00.462195 7222 tablet_replica.cc:442] T fb80a922687d4aa98c794c22b20d4535 P 20432b1f387d4877b8535b0e6e7aab4c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:00.463079 7245 tablet_replica.cc:439] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86bc51927bda4216bb1980a10625cc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } }
I20260620 14:51:00.463181 7245 tablet_replica.cc:442] T fb80a922687d4aa98c794c22b20d4535 P 86bc51927bda4216bb1980a10625cc51: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:51:00.463604 7223 tablet_replica.cc:439] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86bc51927bda4216bb1980a10625cc51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86bc51927bda4216bb1980a10625cc51" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 36149 } } peers { permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 } } peers { permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 } } }
I20260620 14:51:00.463691 7223 tablet_replica.cc:442] T fb80a922687d4aa98c794c22b20d4535 P bd14e3cb98a540ddb4b41d383601ad54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:00.466929 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.463994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.5.118.129:36149)
0620 14:51:00.464036 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:00.464038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:00.464161 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:51:00.464207 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:00.464207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:00.464218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:00.464266 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:00.464392 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.464860 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.466219 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.466437 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.466502 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.466661 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:00.466690 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:00.466728 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:00.466808 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:51:00.466830 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:51:00.472879 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.470151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:38937 (local address 127.5.118.129:36149)
0620 14:51:00.470192 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:00.470193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:00.470327 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:51:00.470367 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:00.470368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:00.470379 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:00.470445 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:00.470539 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.471298 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.472043 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.472382 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.472454 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.472608 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:00.472641 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:00.472696 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:00.472738 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:51:00.472759 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:51:00.472980 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.470080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:36149 (local address 127.5.118.129:38937)
0620 14:51:00.470239 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.470249 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:00.470326 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:00.470448 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:00.470449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:00.470450 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:51:00.470502 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.470504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.471383 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.471384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.471968 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:00.471970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.472453 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.472454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:00.472517 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.472532 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:51:00.472698 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:51:00.472699 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:51:00.472730 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 14:51:00.472761 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260620 14:51:00.475340 7005 raft_consensus.cc:1261] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54 [term 1 FOLLOWER]: Refusing update from remote peer 86bc51927bda4216bb1980a10625cc51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:00.475339 7146 raft_consensus.cc:1261] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c [term 1 FOLLOWER]: Refusing update from remote peer 86bc51927bda4216bb1980a10625cc51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:00.475562 7221 consensus_queue.cc:1035] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd14e3cb98a540ddb4b41d383601ad54" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:00.475692 7221 consensus_queue.cc:1035] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20432b1f387d4877b8535b0e6e7aab4c" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:00.481420 7229 mvcc.cc:205] Tried to move back new op lower bound from 7298937079705587712 to 7298937078254993408. Current Snapshot: MvccSnapshot[applied={T|T < 7298937079705587712}]
I20260620 14:51:00.493144 6708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:00.493253 6708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:00.493916 6708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fd8ad2718624535be756a4ff71e9f5c: Sending DeleteTablet for 3 replicas of tablet b505c4ff20eb4b61b0e2ab6f4dc02508
I20260620 14:51:00.494225 7126 tablet_service.cc:1514] Processing DeleteTablet for tablet b505c4ff20eb4b61b0e2ab6f4dc02508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:00 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:50050
I20260620 14:51:00.494292 6986 tablet_service.cc:1514] Processing DeleteTablet for tablet b505c4ff20eb4b61b0e2ab6f4dc02508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:00 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:36630
I20260620 14:51:00.494369 7237 tablet_replica.cc:330] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c: stopping tablet replica
I20260620 14:51:00.494402 7238 tablet_replica.cc:330] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54: stopping tablet replica
I20260620 14:51:00.494405 6845 tablet_service.cc:1514] Processing DeleteTablet for tablet b505c4ff20eb4b61b0e2ab6f4dc02508 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:00 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:54436
I20260620 14:51:00.494450 7238 raft_consensus.cc:2229] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:00.494448 7237 raft_consensus.cc:2229] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:00.494488 7238 raft_consensus.cc:2258] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:00.494490 7237 raft_consensus.cc:2258] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:00.494496 7239 tablet_replica.cc:330] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51: stopping tablet replica
I20260620 14:51:00.494539 7239 raft_consensus.cc:2229] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:00.494634 7239 raft_consensus.cc:2258] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:00.494892 7237 ts_tablet_manager.cc:1905] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:00.494904 7238 ts_tablet_manager.cc:1905] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:00.496577 7237 ts_tablet_manager.cc:1918] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:00.496619 7237 log.cc:1196] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/b505c4ff20eb4b61b0e2ab6f4dc02508
I20260620 14:51:00.496663 7238 ts_tablet_manager.cc:1918] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:00.496728 7238 log.cc:1196] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/b505c4ff20eb4b61b0e2ab6f4dc02508
I20260620 14:51:00.496807 7237 ts_tablet_manager.cc:1939] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 20432b1f387d4877b8535b0e6e7aab4c: Deleting consensus metadata
I20260620 14:51:00.496945 7238 ts_tablet_manager.cc:1939] T b505c4ff20eb4b61b0e2ab6f4dc02508 P bd14e3cb98a540ddb4b41d383601ad54: Deleting consensus metadata
I20260620 14:51:00.497112 6695 catalog_manager.cc:4841] TS 20432b1f387d4877b8535b0e6e7aab4c (127.5.118.131:46463): tablet b505c4ff20eb4b61b0e2ab6f4dc02508 (table test-table [id=3c047e095a29468690a188fafa686a6d]) successfully deleted
I20260620 14:51:00.497442 7239 ts_tablet_manager.cc:1905] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:00.498970 7239 ts_tablet_manager.cc:1918] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:00.499037 7239 log.cc:1196] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967053866185-5594-0/minicluster-data/ts-0/wal/wals/b505c4ff20eb4b61b0e2ab6f4dc02508
I20260620 14:51:00.499243 7239 ts_tablet_manager.cc:1939] T b505c4ff20eb4b61b0e2ab6f4dc02508 P 86bc51927bda4216bb1980a10625cc51: Deleting consensus metadata
I20260620 14:51:00.499702 6696 catalog_manager.cc:4841] TS bd14e3cb98a540ddb4b41d383601ad54 (127.5.118.130:33519): tablet b505c4ff20eb4b61b0e2ab6f4dc02508 (table test-table [id=3c047e095a29468690a188fafa686a6d]) successfully deleted
I20260620 14:51:00.499998 6693 catalog_manager.cc:4841] TS 86bc51927bda4216bb1980a10625cc51 (127.5.118.129:36149): tablet b505c4ff20eb4b61b0e2ab6f4dc02508 (table test-table [id=3c047e095a29468690a188fafa686a6d]) successfully deleted
Jun 20 14:51:00 dist-test-slave-2m5f krb5kdc[6664](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.118.129@KRBTEST.COM
I20260620 14:51:00.509732 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.500208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.5.118.129:36149)
0620 14:51:00.500254 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:00.500257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:00.500391 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:51:00.500468 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:00.500468 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:00.500477 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:00.500525 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:00.501250 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.501714 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.502823 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.503005 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.503065 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.506528 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:00.506540 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:00.506540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:00.506548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:00.508500 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:00.508871 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:00.508880 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:00.508880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:00.508934 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:00.509128 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:00.509129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:00.509129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:00.509280 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:00.509425 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:00.509504 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:51:00.509586 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 14:51:00.509995 6906 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54454
I20260620 14:51:00.516896 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.510990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52350 (local address 127.5.118.190:42351)
0620 14:51:00.511045 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:00.511047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:00.511057 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:00.511130 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:00.511131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:00.511141 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:00.511185 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:00.511678 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.512233 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.512941 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:00.513150 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:00.513230 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:00.514449 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:00.514451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:00.514451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:00.514459 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:00.515758 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:00.516092 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:00.516094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:00.516094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:00.516152 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:00.516395 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:00.516396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:00.516397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:00.516513 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:00.516615 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:00.516708 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:51:00.516789 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 14:51:00.517132 6708 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52350
I20260620 14:51:00.517788 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 6781
I20260620 14:51:00.523409 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 6921
I20260620 14:51:00.528205 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 7061
I20260620 14:51:00.533696 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 6677
2026-06-20T14:51:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2526 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:00 dist-test-slave-2m5f krb5kdc[7275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:00 dist-test-slave-2m5f krb5kdc[7275](info): set up 2 sockets
Jun 20 14:51:00 dist-test-slave-2m5f krb5kdc[7275](info): commencing operation
krb5kdc: starting...
W20260620 14:51:01.592621 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:01 dist-test-slave-2m5f krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:51:01.724390 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:38615
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:01.806842 7288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:01.807109 7288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:01.807148 7288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:01.808369 7288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:01.808441 7288 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:51:01.808462 7288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:01.808485 7288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:01.808513 7288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:01.808543 7288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:01.810076 7288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:38615
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:01.810385 7288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:01.810616 7288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:01.815609 7294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:01.815594 7293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:01.815590 7296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:01.815958 7288 server_base.cc:1034] running on GCE node
I20260620 14:51:01.816247 7288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:01.816556 7288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:01.817732 7288 hybrid_clock.cc:648] HybridClock initialized: now 1781967061817715 us; error 33 us; skew 500 ppm
Jun 20 14:51:01 dist-test-slave-2m5f krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967061, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:01.820533 7288 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:51:01.821003 7288 webserver.cc:460] Webserver started at http://127.5.118.190:33589/ using document root <none> and password file <none>
I20260620 14:51:01.821266 7288 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:01.821339 7288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:01.821481 7288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:01.822556 7288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "f3b0803a8f65433ebc128f788f7b62b7"
format_stamp: "Formatted at 2026-06-20 14:51:01 on dist-test-slave-2m5f"
server_key: "bfbd1fb482807ad24c19733e1b8474e0"
server_key_iv: "e50aa7b2b3b49a8fde6fc4858c733aa6"
server_key_version: "encryptionkey@0"
I20260620 14:51:01.822923 7288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "f3b0803a8f65433ebc128f788f7b62b7"
format_stamp: "Formatted at 2026-06-20 14:51:01 on dist-test-slave-2m5f"
server_key: "bfbd1fb482807ad24c19733e1b8474e0"
server_key_iv: "e50aa7b2b3b49a8fde6fc4858c733aa6"
server_key_version: "encryptionkey@0"
I20260620 14:51:01.824361 7288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:01.825170 7303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:01.825366 7288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:01.825462 7288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "f3b0803a8f65433ebc128f788f7b62b7"
format_stamp: "Formatted at 2026-06-20 14:51:01 on dist-test-slave-2m5f"
server_key: "bfbd1fb482807ad24c19733e1b8474e0"
server_key_iv: "e50aa7b2b3b49a8fde6fc4858c733aa6"
server_key_version: "encryptionkey@0"
I20260620 14:51:01.825541 7288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:01.844559 7288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:01.846964 7288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:01.847131 7288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:01.851991 7288 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:38615
I20260620 14:51:01.852104 7365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:38615 every 8 connection(s)
I20260620 14:51:01.852432 7288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:01.853149 7366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:01.855731 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7: Bootstrap starting.
I20260620 14:51:01.856345 7366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:01.856603 7366 log.cc:824] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:01.857316 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7: No bootstrap required, opened a new log
I20260620 14:51:01.859694 7366 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b0803a8f65433ebc128f788f7b62b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38615 } }
I20260620 14:51:01.859877 7366 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:01.859843 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 7288
I20260620 14:51:01.859903 7366 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b0803a8f65433ebc128f788f7b62b7, State: Initialized, Role: FOLLOWER
I20260620 14:51:01.859954 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:01.860078 7366 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b0803a8f65433ebc128f788f7b62b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38615 } }
I20260620 14:51:01.860203 5594 external_mini_cluster.cc:1428] Setting key 9597359ea8aa50f86633591431ae5eca
I20260620 14:51:01.860240 7366 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:01.860291 7366 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:01.860351 7366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:01.861256 7366 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b0803a8f65433ebc128f788f7b62b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38615 } }
I20260620 14:51:01.861371 7366 leader_election.cc:302] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3b0803a8f65433ebc128f788f7b62b7; no voters:
I20260620 14:51:01.861548 7366 leader_election.cc:288] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:01.861655 7371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:01.861830 7366 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:01.861905 7371 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [term 1 LEADER]: Becoming Leader. State: Replica: f3b0803a8f65433ebc128f788f7b62b7, State: Running, Role: LEADER
I20260620 14:51:01.862010 7371 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b0803a8f65433ebc128f788f7b62b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38615 } }
I20260620 14:51:01.862468 7372 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3b0803a8f65433ebc128f788f7b62b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b0803a8f65433ebc128f788f7b62b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38615 } } }
I20260620 14:51:01.862586 7372 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:01.862746 7373 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3b0803a8f65433ebc128f788f7b62b7. Latest consensus state: current_term: 1 leader_uuid: "f3b0803a8f65433ebc128f788f7b62b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b0803a8f65433ebc128f788f7b62b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38615 } } }
I20260620 14:51:01.862810 7373 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:01.862888 7380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:01.863443 7380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:01.865214 7380 catalog_manager.cc:1347] Generated new cluster ID: 4ba28e9808cd40a7be8ac9c97f6e32b3
I20260620 14:51:01.865276 7380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:51:01 dist-test-slave-2m5f krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:01.873252 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:01.860842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.5.118.190:38615)
0620 14:51:01.861077 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:01.861088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:01.861127 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:51:01.861289 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:01.861298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:01.861362 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:01.861535 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:01.862415 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:01.863903 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:01.864905 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:01.865521 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:01.865618 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:01.868514 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:01.868519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:01.868519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:01.868531 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:01.871254 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:01.871632 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:01.871635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:01.871635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:01.871744 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:01.871923 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:01.871924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:01.871924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:01.872241 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:01.872382 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:01.872875 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 14:51:01.873012 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":108,"threads_started":1}
I20260620 14:51:01.881199 7380 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:01.881863 7380 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:01.890345 7380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7: Generated new TSK 0
I20260620 14:51:01.890571 7380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:51:01.922847 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38615
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:02.006861 7392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:02.007130 7392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:02.007175 7392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:02.008445 7392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:02.008518 7392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:02.008553 7392 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:51:02.008607 7392 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:51:02.008648 7392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:02.010310 7392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.118.190:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:02.010697 7392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:02.010983 7392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:02.016525 7398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:02.016506 7397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:02.016511 7400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:02.016547 7392 server_base.cc:1034] running on GCE node
I20260620 14:51:02.016960 7392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:02.017221 7392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:02.018393 7392 hybrid_clock.cc:648] HybridClock initialized: now 1781967062018376 us; error 38 us; skew 500 ppm
Jun 20 14:51:02 dist-test-slave-2m5f krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:02.021302 7392 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:51:02.021793 7392 webserver.cc:460] Webserver started at http://127.5.118.129:46169/ using document root <none> and password file <none>
I20260620 14:51:02.022014 7392 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:02.022056 7392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:02.022152 7392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:02.023227 7392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "3b12a94c6d7b4afc8375697e988b0334"
format_stamp: "Formatted at 2026-06-20 14:51:02 on dist-test-slave-2m5f"
server_key: "a0ee406ff7409ea535a8377f913355f2"
server_key_iv: "e815e8d47e703f46fecda93acd819410"
server_key_version: "encryptionkey@0"
I20260620 14:51:02.023538 7392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "3b12a94c6d7b4afc8375697e988b0334"
format_stamp: "Formatted at 2026-06-20 14:51:02 on dist-test-slave-2m5f"
server_key: "a0ee406ff7409ea535a8377f913355f2"
server_key_iv: "e815e8d47e703f46fecda93acd819410"
server_key_version: "encryptionkey@0"
I20260620 14:51:02.024967 7392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:51:02.025686 7407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.025909 7392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:02.025993 7392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "3b12a94c6d7b4afc8375697e988b0334"
format_stamp: "Formatted at 2026-06-20 14:51:02 on dist-test-slave-2m5f"
server_key: "a0ee406ff7409ea535a8377f913355f2"
server_key_iv: "e815e8d47e703f46fecda93acd819410"
server_key_version: "encryptionkey@0"
I20260620 14:51:02.026050 7392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:02.039822 7392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:02.042356 7392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:02.042477 7392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:02.042935 7392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:02.042970 7392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.042999 7392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:02.043066 7392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.050297 7392 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:39987
I20260620 14:51:02.050379 7522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:39987 every 8 connection(s)
I20260620 14:51:02.050740 7392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:51:02 dist-test-slave-2m5f krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:02.054664 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.043488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:49957 (local address 127.5.118.190:38615)
0620 14:51:02.043533 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.043535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.043791 (+ 256us) server_negotiation.cc:407] Connection header received
0620 14:51:02.044460 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.044462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.044481 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.044532 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:02.045399 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.046257 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.048346 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.048549 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.048611 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.051580 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:02.051582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:02.051582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:02.051590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:02.053221 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.053658 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.053660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.053660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.053730 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.054003 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.054004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.054005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.054127 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:02.054216 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.054505 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.054597 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:51:02.054874 7416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.043491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38615 (local address 127.5.118.129:49957)
0620 14:51:02.043672 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.043708 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.044346 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.044611 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.044616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.044992 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:02.045285 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.045293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.046369 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.046371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.048097 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.048101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.048645 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.048647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.048820 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.049480 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:02.049491 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:02.051474 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:02.053317 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.053323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.053325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.053567 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.053811 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.053812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.053812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.053922 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.054215 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:02.054220 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:02.054347 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.054440 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.054573 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260620 14:51:02.055483 7523 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38615
I20260620 14:51:02.055593 7523 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:02.055816 7523 heartbeater.cc:511] Master 127.5.118.190:38615 requested a full tablet report, sending...
I20260620 14:51:02.056553 7320 ts_manager.cc:194] Registered new tserver with Master: 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129:39987)
I20260620 14:51:02.057245 7320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:49957
I20260620 14:51:02.058334 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 7392
I20260620 14:51:02.058451 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:02.058636 5594 external_mini_cluster.cc:1428] Setting key 8ac46a45dd6ab48f1f821d55bb197fd8
I20260620 14:51:02.063165 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.056537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38615 (local address 127.5.118.129:51209)
0620 14:51:02.056724 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.056733 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.056804 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.056934 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.056935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.057114 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:02.057164 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.057165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.057949 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.057950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.058557 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.058559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.059036 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.059037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.059114 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.059591 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:02.059597 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:02.060018 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:02.061780 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.061781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.061782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.062034 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.062350 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.062351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.062351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.062421 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.062797 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:02.062799 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:02.062865 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.062937 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.063083 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260620 14:51:02.063235 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.056623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:51209 (local address 127.5.118.190:38615)
0620 14:51:02.056666 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.056668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.056790 (+ 122us) server_negotiation.cc:407] Connection header received
0620 14:51:02.056854 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.056855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.056866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.056921 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:02.057245 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.057858 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.058654 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.058899 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.058982 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.060118 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:02.060119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:02.060119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:02.060125 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:02.061691 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.062119 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.062121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.062121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.062183 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.062485 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.062486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.062486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.062649 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:02.062790 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.062981 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.063089 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:51:02.087816 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38615
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:02.182367 7532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:02.182727 7532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:02.182791 7532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:02.184238 7532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:02.184319 7532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:02.184358 7532 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:51:02.184392 7532 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:51:02.184438 7532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:02.186158 7532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.118.190:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:02.186475 7532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:02.186760 7532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:02.192502 7540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:02.192498 7538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:02.192505 7537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:02.192647 7532 server_base.cc:1034] running on GCE node
I20260620 14:51:02.193014 7532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:02.193320 7532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:02.194478 7532 hybrid_clock.cc:648] HybridClock initialized: now 1781967062194462 us; error 35 us; skew 500 ppm
Jun 20 14:51:02 dist-test-slave-2m5f krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:02.197327 7532 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:51:02.197810 7532 webserver.cc:460] Webserver started at http://127.5.118.130:46223/ using document root <none> and password file <none>
I20260620 14:51:02.198067 7532 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:02.198134 7532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:02.198282 7532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:02.199520 7532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "ec5ba2bd55eb4202af4d804320aa62f5"
format_stamp: "Formatted at 2026-06-20 14:51:02 on dist-test-slave-2m5f"
server_key: "24ed601a01f820b2edb8bb849514c802"
server_key_iv: "04d926dcfd80e81b3ba095142d2e3bb5"
server_key_version: "encryptionkey@0"
I20260620 14:51:02.199899 7532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "ec5ba2bd55eb4202af4d804320aa62f5"
format_stamp: "Formatted at 2026-06-20 14:51:02 on dist-test-slave-2m5f"
server_key: "24ed601a01f820b2edb8bb849514c802"
server_key_iv: "04d926dcfd80e81b3ba095142d2e3bb5"
server_key_version: "encryptionkey@0"
I20260620 14:51:02.201377 7532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:02.202378 7547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.202704 7532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:02.202838 7532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "ec5ba2bd55eb4202af4d804320aa62f5"
format_stamp: "Formatted at 2026-06-20 14:51:02 on dist-test-slave-2m5f"
server_key: "24ed601a01f820b2edb8bb849514c802"
server_key_iv: "04d926dcfd80e81b3ba095142d2e3bb5"
server_key_version: "encryptionkey@0"
I20260620 14:51:02.202929 7532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:02.213526 7532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:02.216465 7532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:02.216645 7532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:02.217202 7532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:02.217274 7532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.217326 7532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:02.217353 7532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.224483 7532 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:37965
I20260620 14:51:02.224563 7662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:37965 every 8 connection(s)
Jun 20 14:51:02 dist-test-slave-2m5f krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:02.224920 7532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:02.228081 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.217744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:41645 (local address 127.5.118.190:38615)
0620 14:51:02.217867 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.217869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.218037 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:51:02.218641 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.218643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.218659 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.218742 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:02.219637 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.220533 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.222047 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.222406 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.222504 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.225232 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:02.225233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:02.225234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:02.225241 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:02.226683 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.227105 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.227106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.227106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.227172 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.227423 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.227424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.227424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.227544 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:02.227615 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.227945 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.228017 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 14:51:02.228247 7556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.217731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38615 (local address 127.5.118.130:41645)
0620 14:51:02.217911 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.217957 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.218502 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.218823 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.218827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.219203 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:02.219508 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.219515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.220629 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.220630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.221964 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.221967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.222528 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.222529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.222738 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.223445 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:02.223455 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:02.225127 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:02.226790 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.226797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.226799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.227021 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.227244 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.227245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.227245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.227354 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.227606 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:02.227612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:02.227752 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.227891 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.228032 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260620 14:51:02.228781 7663 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38615
I20260620 14:51:02.228868 7663 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:02.229099 7663 heartbeater.cc:511] Master 127.5.118.190:38615 requested a full tablet report, sending...
I20260620 14:51:02.229485 7320 ts_manager.cc:194] Registered new tserver with Master: ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130:37965)
I20260620 14:51:02.229876 7320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:41645
I20260620 14:51:02.233444 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 7532
I20260620 14:51:02.233541 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:02.233839 5594 external_mini_cluster.cc:1428] Setting key 0ec74a302bd20a98c79291aebf3ee228
I20260620 14:51:02.236232 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.230200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:33099 (local address 127.5.118.190:38615)
0620 14:51:02.230243 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.230245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.230839 (+ 594us) server_negotiation.cc:407] Connection header received
0620 14:51:02.230859 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.230860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.230869 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.230920 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:02.231203 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.231642 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.232400 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.232602 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.232750 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.233642 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:02.233644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:02.233644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:02.233651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:02.235074 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.235417 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.235418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.235419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.235477 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.235667 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.235668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.235668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.235818 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:02.235893 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.236041 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.236118 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:51:02.236248 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.230135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38615 (local address 127.5.118.130:33099)
0620 14:51:02.230686 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.230696 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.230746 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.230940 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.230941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.231095 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:02.231143 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.231144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.231722 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.231723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.232319 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.232321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.232677 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.232677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.232734 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.233183 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:02.233188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:02.233550 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:02.235159 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.235160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.235160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.235333 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.235541 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.235541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.235542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.235600 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.235885 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:02.235886 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:02.235934 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.236017 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.236129 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":488,"threads_started":1}
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:51:02.259552 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38615
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:02.343098 7672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:02.343312 7672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:02.343338 7672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:02.344651 7672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:02.344695 7672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:02.344717 7672 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:51:02.344738 7672 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:51:02.344784 7672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:02.346261 7672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.118.190:38615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:02.346516 7672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:02.346760 7672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:02.352223 7678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:02.352272 7677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:02.352245 7680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:02.352443 7672 server_base.cc:1034] running on GCE node
I20260620 14:51:02.352722 7672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:02.352974 7672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:02.354128 7672 hybrid_clock.cc:648] HybridClock initialized: now 1781967062354116 us; error 34 us; skew 500 ppm
Jun 20 14:51:02 dist-test-slave-2m5f krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:02.357146 7672 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:51:02.357682 7672 webserver.cc:460] Webserver started at http://127.5.118.131:45991/ using document root <none> and password file <none>
I20260620 14:51:02.357966 7672 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:02.358042 7672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:02.358217 7672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:02.359459 7672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9"
format_stamp: "Formatted at 2026-06-20 14:51:02 on dist-test-slave-2m5f"
server_key: "4aba6efd89cb0f03971595ca80c744e8"
server_key_iv: "784fd29787efe30beffe39b2e3e9dfd1"
server_key_version: "encryptionkey@0"
I20260620 14:51:02.359863 7672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9"
format_stamp: "Formatted at 2026-06-20 14:51:02 on dist-test-slave-2m5f"
server_key: "4aba6efd89cb0f03971595ca80c744e8"
server_key_iv: "784fd29787efe30beffe39b2e3e9dfd1"
server_key_version: "encryptionkey@0"
I20260620 14:51:02.361469 7672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:02.362413 7687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.362681 7672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:02.362810 7672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9"
format_stamp: "Formatted at 2026-06-20 14:51:02 on dist-test-slave-2m5f"
server_key: "4aba6efd89cb0f03971595ca80c744e8"
server_key_iv: "784fd29787efe30beffe39b2e3e9dfd1"
server_key_version: "encryptionkey@0"
I20260620 14:51:02.362888 7672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:02.379336 7672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:02.381692 7672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:02.381830 7672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:02.382340 7672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:02.382418 7672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.382462 7672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:02.382503 7672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.390321 7672 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:42413
I20260620 14:51:02.390470 7802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:42413 every 8 connection(s)
I20260620 14:51:02.390822 7672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:51:02 dist-test-slave-2m5f krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967062, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:02.394599 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.382937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:46187 (local address 127.5.118.190:38615)
0620 14:51:02.383009 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.383012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.383148 (+ 136us) server_negotiation.cc:407] Connection header received
0620 14:51:02.383687 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.383688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.383704 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.383798 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:02.384772 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.385606 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.387616 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.387949 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.388029 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.391641 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:02.391643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:02.391643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:02.391651 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:02.393256 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.393683 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.393684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.393684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.393746 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.394008 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.394008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.394009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.394126 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:02.394179 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.394458 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.394535 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:51:02.394752 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.382842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38615 (local address 127.5.118.131:46187)
0620 14:51:02.383008 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.383050 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.383574 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.383872 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.383876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.384347 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:02.384664 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.384672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.385733 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.385735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.387489 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.387493 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.388058 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.388059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.388275 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.389106 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:02.389120 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:02.391537 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:02.393341 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.393347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.393348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.393584 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.393847 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.393847 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.393848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.393951 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.394178 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:02.394182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:02.394309 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.394395 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.394523 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260620 14:51:02.394985 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 7672
I20260620 14:51:02.395084 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:02.395294 5594 external_mini_cluster.cc:1428] Setting key 609044d7a3e12529bd3fbfe0aaed6ec2
I20260620 14:51:02.395366 7803 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38615
I20260620 14:51:02.395452 7803 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:02.395658 7803 heartbeater.cc:511] Master 127.5.118.190:38615 requested a full tablet report, sending...
I20260620 14:51:02.396054 7319 ts_manager.cc:194] Registered new tserver with Master: 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413)
I20260620 14:51:02.396379 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:51:02.396579 7319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:46187
I20260620 14:51:02.402215 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.396553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:41205 (local address 127.5.118.190:38615)
0620 14:51:02.396593 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.396595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.396715 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:51:02.396829 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.396830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.396839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.396868 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:02.397234 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.397743 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.398498 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.398707 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.398778 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.399619 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:02.399620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:02.399621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:02.399628 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:02.401104 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.401454 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.401455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.401455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.401512 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.401698 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.401699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.401699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.401813 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:02.401885 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.402033 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.402114 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:51:02.402215 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.396511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38615 (local address 127.5.118.131:41205)
0620 14:51:02.396660 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.396671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.396740 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.396938 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.396939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.397099 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:02.397149 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.397150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.397835 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.397836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.398422 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.398423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.398776 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.398777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.398858 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.399261 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:02.399266 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:02.399532 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:02.401202 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.401203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.401203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.401375 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.401581 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.401581 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.401582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.401640 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.401876 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:02.401877 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:02.401925 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.402017 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.402115 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
Jun 20 14:51:02 dist-test-slave-2m5f krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:51:02 dist-test-slave-2m5f krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:02.420419 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.412526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.5.118.190:38615)
0620 14:51:02.412607 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.412610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.412620 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:02.412675 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.412676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.412799 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.412861 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:02.413472 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.413963 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.414724 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.414929 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.414998 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.417685 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:02.417688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:02.417688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:02.417697 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:02.419187 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.419552 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.419554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.419554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.419615 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.419828 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.419829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.419829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.419956 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:02.420057 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.420231 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.420313 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:51:02.421993 7319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:02.423527 7319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:02.431203 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.426180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39987 (local address 127.0.0.1:36320)
0620 14:51:02.426617 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.426635 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.426693 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.427028 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.427031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.427034 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:02.427115 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.427117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.428731 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.428732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.429836 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.429839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.430834 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.430835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.430932 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.430948 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.431033 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.431092 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":4,"thread_start_us":67,"threads_started":1}
I20260620 14:51:02.431517 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.426324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.5.118.130:37965)
0620 14:51:02.426716 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.426722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.426736 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:51:02.426935 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.426938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.426955 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.427033 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:02.427240 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.428320 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.430312 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.430949 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.431033 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.431093 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.431378 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.431419 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":113,"threads_started":1}
I20260620 14:51:02.431648 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.426513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.5.118.131:42413)
0620 14:51:02.426689 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.426775 (+ 86us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.426937 (+ 162us) server_negotiation.cc:407] Connection header received
0620 14:51:02.426985 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.426987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.427005 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.427082 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:02.427315 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.428585 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.430295 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.431134 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.431238 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.431297 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.431496 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.431539 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260620 14:51:02.431330 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.426180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:37965 (local address 127.0.0.1:48688)
0620 14:51:02.426555 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.426607 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.426685 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.427114 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.427114 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.427117 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:02.427171 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.427173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.428424 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.428425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.430204 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.430207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.431044 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.431045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.431119 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.431123 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.431193 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.431241 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":56,"thread_start_us":85,"threads_started":1}
I20260620 14:51:02.431569 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.426406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:42413 (local address 127.0.0.1:37552)
0620 14:51:02.426819 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.426828 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.426876 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.427156 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.427157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.427159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:02.427243 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.427245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.428667 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.428668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.430204 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.430207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.431336 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.431337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.431404 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.431407 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.431468 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.431507 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":5,"thread_start_us":84,"threads_started":1}
I20260620 14:51:02.432065 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.426409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36320 (local address 127.5.118.129:39987)
0620 14:51:02.426820 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.426824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.426834 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:02.426881 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.426884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.426904 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.427203 (+ 299us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:02.427286 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.428611 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.429959 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.430737 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.430820 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.430882 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.431895 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.431949 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":90,"threads_started":1}
I20260620 14:51:02.432607 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 99a66cbfb848456e9f481e25c3679330 (DEFAULT_TABLE table=test-table [id=431f3acb773d44f1a7cc15c46d269d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:02.432638 7457 tablet_service.cc:1467] Processing CreateTablet for tablet 99a66cbfb848456e9f481e25c3679330 (DEFAULT_TABLE table=test-table [id=431f3acb773d44f1a7cc15c46d269d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:02.432983 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a66cbfb848456e9f481e25c3679330. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:02.432982 7457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a66cbfb848456e9f481e25c3679330. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:02.436295 7827 tablet_bootstrap.cc:492] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: Bootstrap starting.
I20260620 14:51:02.436331 7826 tablet_bootstrap.cc:492] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: Bootstrap starting.
I20260620 14:51:02.436980 7827 tablet_bootstrap.cc:654] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:02.437070 7826 tablet_bootstrap.cc:654] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:02.437084 7737 tablet_service.cc:1467] Processing CreateTablet for tablet 99a66cbfb848456e9f481e25c3679330 (DEFAULT_TABLE table=test-table [id=431f3acb773d44f1a7cc15c46d269d22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:02.437376 7827 log.cc:824] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:02.437436 7826 log.cc:824] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:02.437493 7737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a66cbfb848456e9f481e25c3679330. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:02.438195 7827 tablet_bootstrap.cc:492] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: No bootstrap required, opened a new log
I20260620 14:51:02.438284 7827 ts_tablet_manager.cc:1397] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:51:02.440038 7827 raft_consensus.cc:348] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.440250 7827 raft_consensus.cc:374] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:02.440297 7827 raft_consensus.cc:729] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b12a94c6d7b4afc8375697e988b0334, State: Initialized, Role: FOLLOWER
I20260620 14:51:02.440341 7830 tablet_bootstrap.cc:492] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Bootstrap starting.
I20260620 14:51:02.440444 7827 consensus_queue.cc:260] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.440752 7523 heartbeater.cc:503] Master 127.5.118.190:38615 was elected leader, sending a full tablet report...
I20260620 14:51:02.440744 7827 ts_tablet_manager.cc:1428] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:51:02.441120 7826 tablet_bootstrap.cc:492] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: No bootstrap required, opened a new log
I20260620 14:51:02.441206 7826 ts_tablet_manager.cc:1397] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 14:51:02.442981 7826 raft_consensus.cc:348] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.443142 7826 raft_consensus.cc:374] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:02.443197 7826 raft_consensus.cc:729] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5ba2bd55eb4202af4d804320aa62f5, State: Initialized, Role: FOLLOWER
I20260620 14:51:02.443351 7826 consensus_queue.cc:260] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.443629 7826 ts_tablet_manager.cc:1428] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:51:02.443679 7663 heartbeater.cc:503] Master 127.5.118.190:38615 was elected leader, sending a full tablet report...
I20260620 14:51:02.444464 7830 tablet_bootstrap.cc:654] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:02.444841 7830 log.cc:824] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:02.445551 7830 tablet_bootstrap.cc:492] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: No bootstrap required, opened a new log
I20260620 14:51:02.445638 7830 ts_tablet_manager.cc:1397] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 14:51:02.447184 7830 raft_consensus.cc:348] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.447366 7830 raft_consensus.cc:374] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:02.447417 7830 raft_consensus.cc:729] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cd25e44ae5452f89cbd4e1ed6cbde9, State: Initialized, Role: FOLLOWER
I20260620 14:51:02.447546 7830 consensus_queue.cc:260] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.447916 7830 ts_tablet_manager.cc:1428] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:51:02.447968 7803 heartbeater.cc:503] Master 127.5.118.190:38615 was elected leader, sending a full tablet report...
W20260620 14:51:02.475692 7664 tablet.cc:2369] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:51:02.551638 7524 tablet.cc:2369] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:51:02.641464 7804 tablet.cc:2369] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:02.678239 7832 raft_consensus.cc:482] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:02.678377 7832 raft_consensus.cc:504] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.678857 7832 leader_election.cc:288] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129:39987), 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413)
I20260620 14:51:02.681910 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.679256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:32825 (local address 127.5.118.129:39987)
0620 14:51:02.679309 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.679312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.679323 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:51:02.679355 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.679355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.679363 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.679458 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:02.679625 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.680117 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.681157 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.681539 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.681632 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.681642 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.681796 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.681842 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:51:02.681916 7556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.679095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39987 (local address 127.5.118.130:32825)
0620 14:51:02.679158 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.679172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.679223 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.679467 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.679469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.679471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:02.679541 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.679543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.680254 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.680255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.681061 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.681063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.681627 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.681628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.681722 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.681726 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.681790 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.681832 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":17}
I20260620 14:51:02.681924 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.679393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:45433 (local address 127.5.118.131:42413)
0620 14:51:02.679441 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.679444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.679494 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:51:02.679579 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.679580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.679587 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.679648 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:02.679810 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.680306 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.681226 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.681577 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.681664 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.681673 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.681822 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.681859 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:51:02.681934 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.679255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:42413 (local address 127.5.118.130:45433)
0620 14:51:02.679436 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.679448 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.679505 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.679667 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.679668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.679669 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:02.679740 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.679742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.680388 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.680389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.681118 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.681120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.681665 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.681666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.681743 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.681746 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.681804 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.681845 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":117,"threads_started":1}
I20260620 14:51:02.682328 7477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a66cbfb848456e9f481e25c3679330" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b12a94c6d7b4afc8375697e988b0334" is_pre_election: true
I20260620 14:51:02.682361 7757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a66cbfb848456e9f481e25c3679330" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" is_pre_election: true
I20260620 14:51:02.682519 7757 raft_consensus.cc:2454] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 0.
I20260620 14:51:02.682520 7477 raft_consensus.cc:2454] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 0.
I20260620 14:51:02.682785 7551 leader_election.cc:302] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b12a94c6d7b4afc8375697e988b0334, ec5ba2bd55eb4202af4d804320aa62f5; no voters:
I20260620 14:51:02.682938 7832 raft_consensus.cc:2781] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:02.683017 7832 raft_consensus.cc:482] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:02.683035 7832 raft_consensus.cc:3037] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:02.683784 7832 raft_consensus.cc:504] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.683893 7832 leader_election.cc:288] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 election: Requested vote from peers 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129:39987), 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413)
I20260620 14:51:02.684126 7757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a66cbfb848456e9f481e25c3679330" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9"
I20260620 14:51:02.684104 7477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a66cbfb848456e9f481e25c3679330" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b12a94c6d7b4afc8375697e988b0334"
I20260620 14:51:02.684243 7757 raft_consensus.cc:3037] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:02.684249 7477 raft_consensus.cc:3037] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:02.684917 7757 raft_consensus.cc:2454] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 1.
I20260620 14:51:02.685158 7551 leader_election.cc:302] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79cd25e44ae5452f89cbd4e1ed6cbde9, ec5ba2bd55eb4202af4d804320aa62f5; no voters:
I20260620 14:51:02.685194 7477 raft_consensus.cc:2454] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 1.
I20260620 14:51:02.685388 7832 raft_consensus.cc:2781] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:02.685580 7832 raft_consensus.cc:686] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 LEADER]: Becoming Leader. State: Replica: ec5ba2bd55eb4202af4d804320aa62f5, State: Running, Role: LEADER
I20260620 14:51:02.685688 7832 consensus_queue.cc:237] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.686538 7319 catalog_manager.cc:5495] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130). New cstate: current_term: 1 leader_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:51:02 dist-test-slave-2m5f krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967061, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:02.740490 7370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.733010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.5.118.190:38615)
0620 14:51:02.733101 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.733104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.733213 (+ 109us) server_negotiation.cc:407] Connection header received
0620 14:51:02.733297 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.733298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.733310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.733378 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:02.733864 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.734356 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.735074 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.735258 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.735336 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.737539 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:02.737541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:02.737541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:02.737554 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:02.739183 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.739627 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.739629 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.739629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.739698 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.739892 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.739894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.739894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.740013 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:02.740086 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.740282 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.740373 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:51:02.740568 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.732918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38615 (local address 127.0.0.1:59052)
0620 14:51:02.733131 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.733142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.733235 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.733384 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.733385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.733731 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:02.733777 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.733779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.734444 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.734445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.734988 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.734990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.735341 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.735341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.735409 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.735981 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:02.735985 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:02.737432 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:02.739308 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.739319 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.739322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.739537 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.739764 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:02.739765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:02.739765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:02.739825 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:02.740076 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:02.740085 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:02.740132 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.740255 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.740442 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":72,"thread_start_us":117,"threads_started":1}
I20260620 14:51:02.742631 7319 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:59052:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:51:02.743009 7319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:02.744611 7457 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=949d3e044e6f434c8a48759cbf0e09c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:51:02.744637 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=949d3e044e6f434c8a48759cbf0e09c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:51:02.744655 7737 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=949d3e044e6f434c8a48759cbf0e09c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:51:02.744807 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd8fd70e2bf4c4ebf941c3cdab9a0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:02.744807 7457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd8fd70e2bf4c4ebf941c3cdab9a0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:02.744818 7737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd8fd70e2bf4c4ebf941c3cdab9a0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:02.746071 7830 tablet_bootstrap.cc:492] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Bootstrap starting.
I20260620 14:51:02.746315 7826 tablet_bootstrap.cc:492] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5: Bootstrap starting.
I20260620 14:51:02.746704 7830 tablet_bootstrap.cc:654] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:02.746714 7826 tablet_bootstrap.cc:654] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:02.747352 7827 tablet_bootstrap.cc:492] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: Bootstrap starting.
I20260620 14:51:02.747521 7830 tablet_bootstrap.cc:492] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9: No bootstrap required, opened a new log
I20260620 14:51:02.747578 7830 ts_tablet_manager.cc:1397] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:02.747642 7826 tablet_bootstrap.cc:492] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5: No bootstrap required, opened a new log
I20260620 14:51:02.747691 7826 ts_tablet_manager.cc:1397] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:02.747736 7830 raft_consensus.cc:348] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.747784 7827 tablet_bootstrap.cc:654] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:02.747813 7830 raft_consensus.cc:374] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:02.747834 7830 raft_consensus.cc:729] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cd25e44ae5452f89cbd4e1ed6cbde9, State: Initialized, Role: FOLLOWER
I20260620 14:51:02.747824 7826 raft_consensus.cc:348] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.747892 7826 raft_consensus.cc:374] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:02.747915 7826 raft_consensus.cc:729] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5ba2bd55eb4202af4d804320aa62f5, State: Initialized, Role: FOLLOWER
I20260620 14:51:02.747890 7830 consensus_queue.cc:260] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.747958 7826 consensus_queue.cc:260] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.747983 7830 ts_tablet_manager.cc:1428] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.748044 7826 ts_tablet_manager.cc:1428] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.748117 7834 tablet_replica.cc:439] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } } }
I20260620 14:51:02.748235 7834 tablet_replica.cc:442] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:02.748415 7832 tablet_replica.cc:439] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } } }
I20260620 14:51:02.748507 7832 tablet_replica.cc:442] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:02.749336 7827 tablet_bootstrap.cc:492] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: No bootstrap required, opened a new log
I20260620 14:51:02.749380 7827 ts_tablet_manager.cc:1397] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:51:02.749540 7827 raft_consensus.cc:348] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.749598 7827 raft_consensus.cc:374] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:02.749614 7827 raft_consensus.cc:729] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b12a94c6d7b4afc8375697e988b0334, State: Initialized, Role: FOLLOWER
I20260620 14:51:02.749665 7827 consensus_queue.cc:260] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.749764 7827 ts_tablet_manager.cc:1428] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:02.749914 7831 tablet_replica.cc:439] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } } }
I20260620 14:51:02.750021 7831 tablet_replica.cc:442] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:02.949512 7831 raft_consensus.cc:482] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:02.949612 7831 raft_consensus.cc:504] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.950042 7831 leader_election.cc:288] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130:37965), 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413)
I20260620 14:51:02.954517 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.950328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:53869 (local address 127.5.118.131:42413)
0620 14:51:02.950381 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.950384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.950480 (+ 96us) server_negotiation.cc:407] Connection header received
0620 14:51:02.950548 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.950549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.950560 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.950646 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:02.950967 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.951747 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.953428 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.953992 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.954081 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.954091 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.954415 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.954460 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:51:02.954564 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.950265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:42413 (local address 127.5.118.129:53869)
0620 14:51:02.950395 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.950408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.950487 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.950726 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.950727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.950730 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:02.950821 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.950824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.951862 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.951864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.953312 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.953316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.954153 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.954155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.954290 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.954297 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.954387 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.954447 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":68,"thread_start_us":64,"threads_started":1}
I20260620 14:51:02.954723 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.950265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:37965 (local address 127.5.118.129:46927)
0620 14:51:02.950396 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.950408 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.950466 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.950796 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.950797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.950799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:02.950855 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.950857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.951862 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.951864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.953312 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.953316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.954462 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.954463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.954528 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.954531 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.954596 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.954635 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":79,"thread_start_us":60,"threads_started":1}
I20260620 14:51:02.954886 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.950385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:46927 (local address 127.5.118.130:37965)
0620 14:51:02.950533 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.950536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.950542 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:51:02.950566 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.950566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.950576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.950669 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:02.950967 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.951735 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.953428 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.953993 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.954082 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.954091 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.954642 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.954813 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260620 14:51:02.954917 7757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd8fd70e2bf4c4ebf941c3cdab9a0b3" candidate_uuid: "3b12a94c6d7b4afc8375697e988b0334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" is_pre_election: true
I20260620 14:51:02.955019 7757 raft_consensus.cc:2454] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b12a94c6d7b4afc8375697e988b0334 in term 0.
I20260620 14:51:02.955258 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd8fd70e2bf4c4ebf941c3cdab9a0b3" candidate_uuid: "3b12a94c6d7b4afc8375697e988b0334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" is_pre_election: true
I20260620 14:51:02.955281 7411 leader_election.cc:302] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b12a94c6d7b4afc8375697e988b0334, 79cd25e44ae5452f89cbd4e1ed6cbde9; no voters:
I20260620 14:51:02.955353 7617 raft_consensus.cc:2454] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b12a94c6d7b4afc8375697e988b0334 in term 0.
I20260620 14:51:02.955478 7831 raft_consensus.cc:2781] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:02.955552 7831 raft_consensus.cc:482] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:02.955590 7831 raft_consensus.cc:3037] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:02.956429 7831 raft_consensus.cc:504] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.956600 7831 leader_election.cc:288] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [CANDIDATE]: Term 1 election: Requested vote from peers ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130:37965), 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413)
I20260620 14:51:02.956825 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd8fd70e2bf4c4ebf941c3cdab9a0b3" candidate_uuid: "3b12a94c6d7b4afc8375697e988b0334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5ba2bd55eb4202af4d804320aa62f5"
I20260620 14:51:02.956919 7617 raft_consensus.cc:3037] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:02.956904 7757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd8fd70e2bf4c4ebf941c3cdab9a0b3" candidate_uuid: "3b12a94c6d7b4afc8375697e988b0334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9"
I20260620 14:51:02.956983 7757 raft_consensus.cc:3037] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:02.957748 7757 raft_consensus.cc:2454] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b12a94c6d7b4afc8375697e988b0334 in term 1.
I20260620 14:51:02.957824 7617 raft_consensus.cc:2454] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b12a94c6d7b4afc8375697e988b0334 in term 1.
I20260620 14:51:02.958016 7410 leader_election.cc:302] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b12a94c6d7b4afc8375697e988b0334, ec5ba2bd55eb4202af4d804320aa62f5; no voters:
I20260620 14:51:02.958117 7831 raft_consensus.cc:2781] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:02.958249 7831 raft_consensus.cc:686] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 LEADER]: Becoming Leader. State: Replica: 3b12a94c6d7b4afc8375697e988b0334, State: Running, Role: LEADER
I20260620 14:51:02.958410 7831 consensus_queue.cc:237] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:02.958959 7850 tablet_replica.cc:439] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: TxnStatusTablet state changed. Reason: New leader 3b12a94c6d7b4afc8375697e988b0334. Latest consensus state: current_term: 1 leader_uuid: "3b12a94c6d7b4afc8375697e988b0334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } } }
I20260620 14:51:02.959059 7850 tablet_replica.cc:442] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:51:02.959262 7852 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:51:02.959403 7852 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:51:02.959450 7320 catalog_manager.cc:5495] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129). New cstate: current_term: 1 leader_uuid: "3b12a94c6d7b4afc8375697e988b0334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:02.977941 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.974868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39987 (local address 127.0.0.1:36336)
0620 14:51:02.974952 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.974966 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.975024 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.975227 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.975228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.975229 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:51:02.975276 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.975279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.976262 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.976263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.976889 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.976893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.977383 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.977384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.977437 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.977453 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:51:02.977792 (+ 339us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:51:02.977800 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.977838 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.977880 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:51:02.977955 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.974981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.5.118.129:39987)
0620 14:51:02.975048 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.975050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.975062 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:51:02.975117 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.975118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.975141 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.975217 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:02.975323 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.976108 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.976985 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.977302 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.977374 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.977546 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:02.977677 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:02.977761 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.977844 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.977868 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:51:02.979946 7617 raft_consensus.cc:1261] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Refusing update from remote peer 3b12a94c6d7b4afc8375697e988b0334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:02.979964 7757 raft_consensus.cc:1261] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Refusing update from remote peer 3b12a94c6d7b4afc8375697e988b0334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:02.980337 7850 consensus_queue.cc:1035] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:02.980487 7831 consensus_queue.cc:1035] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:02.981559 7832 tablet_replica.cc:439] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5: TxnStatusTablet state changed. Reason: New leader 3b12a94c6d7b4afc8375697e988b0334. Latest consensus state: current_term: 1 leader_uuid: "3b12a94c6d7b4afc8375697e988b0334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } } }
I20260620 14:51:02.981658 7832 tablet_replica.cc:442] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:02.981626 7834 tablet_replica.cc:439] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9: TxnStatusTablet state changed. Reason: New leader 3b12a94c6d7b4afc8375697e988b0334. Latest consensus state: current_term: 1 leader_uuid: "3b12a94c6d7b4afc8375697e988b0334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } } }
I20260620 14:51:02.981709 7834 tablet_replica.cc:442] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:02.982724 7831 tablet_replica.cc:439] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b12a94c6d7b4afc8375697e988b0334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } } }
I20260620 14:51:02.982820 7831 tablet_replica.cc:442] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:51:02.982884 7855 mvcc.cc:205] Tried to move back new op lower bound from 7298937089962983424 to 7298937089878192128. Current Snapshot: MvccSnapshot[applied={T|T < 7298937089962983424}]
I20260620 14:51:02.982977 7831 tablet_replica.cc:439] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b12a94c6d7b4afc8375697e988b0334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } } }
I20260620 14:51:02.983052 7831 tablet_replica.cc:442] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 3b12a94c6d7b4afc8375697e988b0334: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:51:02.983011 7832 tablet_replica.cc:439] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b12a94c6d7b4afc8375697e988b0334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } } }
I20260620 14:51:02.983088 7832 tablet_replica.cc:442] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P ec5ba2bd55eb4202af4d804320aa62f5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:02.983315 7856 mvcc.cc:205] Tried to move back new op lower bound from 7298937089962983424 to 7298937089878192128. Current Snapshot: MvccSnapshot[applied={T|T < 7298937089962983424}]
I20260620 14:51:02.983418 7834 tablet_replica.cc:439] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b12a94c6d7b4afc8375697e988b0334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } } }
I20260620 14:51:02.983498 7834 tablet_replica.cc:442] T 6fd8fd70e2bf4c4ebf941c3cdab9a0b3 P 79cd25e44ae5452f89cbd4e1ed6cbde9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:02.983700 7857 mvcc.cc:205] Tried to move back new op lower bound from 7298937089962983424 to 7298937089878192128. Current Snapshot: MvccSnapshot[applied={T|T < 7298937089962983424}]
I20260620 14:51:02.988844 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.985939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48694 (local address 127.5.118.130:37965)
0620 14:51:02.985987 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.985990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.986141 (+ 151us) server_negotiation.cc:407] Connection header received
0620 14:51:02.986186 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.986187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.986203 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.986271 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:02.986405 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.986862 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.987981 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.988199 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.988302 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.988534 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:02.988634 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:02.988706 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.988764 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.988791 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:51:02.995144 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.992226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:59865 (local address 127.5.118.129:39987)
0620 14:51:02.992269 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.992271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.992419 (+ 148us) server_negotiation.cc:407] Connection header received
0620 14:51:02.992453 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.992454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.992465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.992523 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:02.992653 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.993394 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.994285 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.994582 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.994655 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.994880 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:02.994913 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:02.994968 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.995048 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.995066 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:51:02.995155 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.992121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39987 (local address 127.5.118.130:59865)
0620 14:51:02.992343 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.992355 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:02.992412 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:02.992531 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:02.992532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:02.992533 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:51:02.992594 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.992595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.993490 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.993490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.994189 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:02.994192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.994672 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.994672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:02.994766 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.994782 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:51:02.994998 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:51:02.995000 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:51:02.995036 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 14:51:02.995077 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":142,"threads_started":1}
I20260620 14:51:02.998068 7757 raft_consensus.cc:1261] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Refusing update from remote peer ec5ba2bd55eb4202af4d804320aa62f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:02.998203 7477 raft_consensus.cc:1261] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Refusing update from remote peer ec5ba2bd55eb4202af4d804320aa62f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:02.998381 7836 consensus_queue.cc:1035] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:02.998476 7832 consensus_queue.cc:1035] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:03.008822 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:03.005371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:54149 (local address 127.5.118.130:37965)
0620 14:51:03.005532 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:03.005534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:03.005542 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:03.005562 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:03.005563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:03.005574 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:03.005653 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:03.005826 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:03.006648 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:03.007931 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:03.008317 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:03.008400 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:03.008570 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:03.008607 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:03.008669 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:03.008713 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:51:03.008737 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260620 14:51:03.008945 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:03.005178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:37965 (local address 127.5.118.129:54149)
0620 14:51:03.005335 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:03.005348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:03.005399 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:03.005657 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:03.005658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:03.005659 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:51:03.005741 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:03.005743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:03.006804 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:03.006805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:03.007835 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:03.007838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:03.008403 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:03.008404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:03.008461 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:03.008478 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:51:03.008670 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:51:03.008671 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:51:03.008701 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 14:51:03.008821 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260620 14:51:03.018446 7320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:03.018574 7320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:03.019446 7320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7: Sending DeleteTablet for 3 replicas of tablet 99a66cbfb848456e9f481e25c3679330
I20260620 14:51:03.019920 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a66cbfb848456e9f481e25c3679330 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:03 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:48688
I20260620 14:51:03.019920 7457 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a66cbfb848456e9f481e25c3679330 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:03 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:36320
I20260620 14:51:03.019979 7320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:03.020131 7320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:03.020287 7737 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a66cbfb848456e9f481e25c3679330 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:03 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37552
I20260620 14:51:03.020520 7878 tablet_replica.cc:330] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: stopping tablet replica
I20260620 14:51:03.020635 7878 raft_consensus.cc:2229] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:03.020689 7878 raft_consensus.cc:2258] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:03.021374 7878 ts_tablet_manager.cc:1905] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:03.021811 7737 tablet_service.cc:1467] Processing CreateTablet for tablet 442b4c4cd9a54f73b935c504f0b2cb41 (DEFAULT_TABLE table=test-table [id=0cb5dfc7ff5c4a3797d65aa3215c81df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:03.021878 7457 tablet_service.cc:1467] Processing CreateTablet for tablet 442b4c4cd9a54f73b935c504f0b2cb41 (DEFAULT_TABLE table=test-table [id=0cb5dfc7ff5c4a3797d65aa3215c81df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:03.021934 7737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442b4c4cd9a54f73b935c504f0b2cb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:03.022020 7457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442b4c4cd9a54f73b935c504f0b2cb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:03.023269 7830 tablet_bootstrap.cc:492] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Bootstrap starting.
I20260620 14:51:03.023427 7827 tablet_bootstrap.cc:492] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334: Bootstrap starting.
I20260620 14:51:03.023478 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 442b4c4cd9a54f73b935c504f0b2cb41 (DEFAULT_TABLE table=test-table [id=0cb5dfc7ff5c4a3797d65aa3215c81df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:03.023590 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442b4c4cd9a54f73b935c504f0b2cb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:03.023662 7830 tablet_bootstrap.cc:654] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:03.023823 7827 tablet_bootstrap.cc:654] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:03.024240 7878 ts_tablet_manager.cc:1918] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:03.024309 7878 log.cc:1196] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/99a66cbfb848456e9f481e25c3679330
I20260620 14:51:03.024567 7878 ts_tablet_manager.cc:1939] T 99a66cbfb848456e9f481e25c3679330 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting consensus metadata
I20260620 14:51:03.024981 7307 catalog_manager.cc:4841] TS 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413): tablet 99a66cbfb848456e9f481e25c3679330 (table test-table [id=431f3acb773d44f1a7cc15c46d269d22]) successfully deleted
I20260620 14:51:03.025086 7830 tablet_bootstrap.cc:492] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9: No bootstrap required, opened a new log
I20260620 14:51:03.025136 7826 tablet_bootstrap.cc:492] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5: Bootstrap starting.
I20260620 14:51:03.025128 7830 ts_tablet_manager.cc:1397] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:03.025281 7830 raft_consensus.cc:348] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:03.025352 7830 raft_consensus.cc:374] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:03.025373 7830 raft_consensus.cc:729] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cd25e44ae5452f89cbd4e1ed6cbde9, State: Initialized, Role: FOLLOWER
I20260620 14:51:03.025417 7830 consensus_queue.cc:260] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:03.025504 7830 ts_tablet_manager.cc:1428] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:03.025615 7826 tablet_bootstrap.cc:654] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:03.026424 7826 tablet_bootstrap.cc:492] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5: No bootstrap required, opened a new log
I20260620 14:51:03.026474 7826 ts_tablet_manager.cc:1397] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:03.026629 7826 raft_consensus.cc:348] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:03.026695 7826 raft_consensus.cc:374] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:03.026713 7826 raft_consensus.cc:729] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5ba2bd55eb4202af4d804320aa62f5, State: Initialized, Role: FOLLOWER
I20260620 14:51:03.026752 7826 consensus_queue.cc:260] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:03.026846 7826 ts_tablet_manager.cc:1428] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:03.028342 7876 tablet_replica.cc:330] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: stopping tablet replica
I20260620 14:51:03.028630 7876 raft_consensus.cc:2229] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:03.028463 7877 tablet_replica.cc:330] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: stopping tablet replica
I20260620 14:51:03.029250 7877 raft_consensus.cc:2229] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:03.029379 7877 raft_consensus.cc:2258] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:03.028745 7876 raft_consensus.cc:2258] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:03.030201 7876 ts_tablet_manager.cc:1905] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:03.031932 7877 ts_tablet_manager.cc:1905] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:03.033015 7827 tablet_bootstrap.cc:492] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334: No bootstrap required, opened a new log
I20260620 14:51:03.033078 7827 ts_tablet_manager.cc:1397] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260620 14:51:03.033245 7827 raft_consensus.cc:348] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:03.033319 7827 raft_consensus.cc:374] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:03.033342 7827 raft_consensus.cc:729] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b12a94c6d7b4afc8375697e988b0334, State: Initialized, Role: FOLLOWER
I20260620 14:51:03.033401 7827 consensus_queue.cc:260] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:03.033501 7827 ts_tablet_manager.cc:1428] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:03.033857 7877 ts_tablet_manager.cc:1918] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:03.033941 7877 log.cc:1196] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/wal/wals/99a66cbfb848456e9f481e25c3679330
I20260620 14:51:03.034229 7876 ts_tablet_manager.cc:1918] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:03.034274 7877 ts_tablet_manager.cc:1939] T 99a66cbfb848456e9f481e25c3679330 P 3b12a94c6d7b4afc8375697e988b0334: Deleting consensus metadata
I20260620 14:51:03.034385 7876 log.cc:1196] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/99a66cbfb848456e9f481e25c3679330
I20260620 14:51:03.034684 7307 catalog_manager.cc:4841] TS 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129:39987): tablet 99a66cbfb848456e9f481e25c3679330 (table test-table [id=431f3acb773d44f1a7cc15c46d269d22]) successfully deleted
I20260620 14:51:03.034806 7876 ts_tablet_manager.cc:1939] T 99a66cbfb848456e9f481e25c3679330 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting consensus metadata
I20260620 14:51:03.035259 7306 catalog_manager.cc:4841] TS ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130:37965): tablet 99a66cbfb848456e9f481e25c3679330 (table test-table [id=431f3acb773d44f1a7cc15c46d269d22]) successfully deleted
I20260620 14:51:03.234350 7850 raft_consensus.cc:482] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:03.234486 7850 raft_consensus.cc:504] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:03.234697 7850 leader_election.cc:288] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130:37965), 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413)
I20260620 14:51:03.235040 7757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b4c4cd9a54f73b935c504f0b2cb41" candidate_uuid: "3b12a94c6d7b4afc8375697e988b0334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" is_pre_election: true
I20260620 14:51:03.235040 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b4c4cd9a54f73b935c504f0b2cb41" candidate_uuid: "3b12a94c6d7b4afc8375697e988b0334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" is_pre_election: true
I20260620 14:51:03.235146 7757 raft_consensus.cc:2454] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b12a94c6d7b4afc8375697e988b0334 in term 0.
I20260620 14:51:03.235146 7617 raft_consensus.cc:2454] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b12a94c6d7b4afc8375697e988b0334 in term 0.
I20260620 14:51:03.235342 7411 leader_election.cc:302] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b12a94c6d7b4afc8375697e988b0334, 79cd25e44ae5452f89cbd4e1ed6cbde9; no voters:
I20260620 14:51:03.235504 7850 raft_consensus.cc:2781] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:03.235589 7850 raft_consensus.cc:482] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:03.235616 7850 raft_consensus.cc:3037] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:03.236459 7850 raft_consensus.cc:504] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:03.236614 7850 leader_election.cc:288] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [CANDIDATE]: Term 1 election: Requested vote from peers ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130:37965), 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413)
I20260620 14:51:03.236845 7617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b4c4cd9a54f73b935c504f0b2cb41" candidate_uuid: "3b12a94c6d7b4afc8375697e988b0334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec5ba2bd55eb4202af4d804320aa62f5"
I20260620 14:51:03.236845 7757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442b4c4cd9a54f73b935c504f0b2cb41" candidate_uuid: "3b12a94c6d7b4afc8375697e988b0334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9"
I20260620 14:51:03.236925 7617 raft_consensus.cc:3037] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:03.236948 7757 raft_consensus.cc:3037] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:03.237567 7617 raft_consensus.cc:2454] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b12a94c6d7b4afc8375697e988b0334 in term 1.
I20260620 14:51:03.237576 7757 raft_consensus.cc:2454] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b12a94c6d7b4afc8375697e988b0334 in term 1.
I20260620 14:51:03.237758 7410 leader_election.cc:302] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b12a94c6d7b4afc8375697e988b0334, ec5ba2bd55eb4202af4d804320aa62f5; no voters:
I20260620 14:51:03.237864 7850 raft_consensus.cc:2781] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:03.237939 7850 raft_consensus.cc:686] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 LEADER]: Becoming Leader. State: Replica: 3b12a94c6d7b4afc8375697e988b0334, State: Running, Role: LEADER
I20260620 14:51:03.237993 7850 consensus_queue.cc:237] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:03.238592 7320 catalog_manager.cc:5495] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129). New cstate: current_term: 1 leader_uuid: "3b12a94c6d7b4afc8375697e988b0334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:03.257339 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:03.254833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.5.118.129:39987)
0620 14:51:03.254910 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:03.254912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:03.254922 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:03.254953 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:03.254954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:03.254962 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:03.255012 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:03.255110 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:03.255645 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:03.256520 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:03.256771 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:03.256836 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:03.257033 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:03.257065 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:03.257120 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:03.257171 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:51:03.257188 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:51:03.262365 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:03.259894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:39599 (local address 127.5.118.129:39987)
0620 14:51:03.259988 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:03.259989 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:03.260005 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:51:03.260024 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:03.260025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:03.260033 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:03.260091 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:03.260249 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:03.260762 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:03.261616 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:03.261915 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:03.261981 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:03.262109 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:03.262136 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:03.262181 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:03.262229 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:51:03.262245 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:51:03.262437 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:03.259819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39987 (local address 127.5.118.129:39599)
0620 14:51:03.259893 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:03.259913 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:03.259960 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:03.260100 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:03.260101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:03.260102 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:51:03.260135 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:03.260136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:03.260836 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:03.260836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:03.261496 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:03.261498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:03.262001 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:03.262002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:03.262043 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:03.262048 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:51:03.262188 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:51:03.262189 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:51:03.262220 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 14:51:03.262263 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:51:03.264678 7757 raft_consensus.cc:1261] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Refusing update from remote peer 3b12a94c6d7b4afc8375697e988b0334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:03.264686 7617 raft_consensus.cc:1261] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Refusing update from remote peer 3b12a94c6d7b4afc8375697e988b0334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:03.264896 7850 consensus_queue.cc:1035] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:03.264994 7831 consensus_queue.cc:1035] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:51:03.269683 7855 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:51:04.789937 7319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:04.790053 7319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:04.791018 7319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7: Sending DeleteTablet for 3 replicas of tablet 442b4c4cd9a54f73b935c504f0b2cb41
I20260620 14:51:04.791316 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 442b4c4cd9a54f73b935c504f0b2cb41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:04 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:48688
I20260620 14:51:04.791375 7737 tablet_service.cc:1514] Processing DeleteTablet for tablet 442b4c4cd9a54f73b935c504f0b2cb41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:04 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37552
I20260620 14:51:04.791492 7457 tablet_service.cc:1514] Processing DeleteTablet for tablet 442b4c4cd9a54f73b935c504f0b2cb41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:04 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:36320
I20260620 14:51:04.791587 7908 tablet_replica.cc:330] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9: stopping tablet replica
I20260620 14:51:04.791595 7907 tablet_replica.cc:330] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5: stopping tablet replica
I20260620 14:51:04.791644 7907 raft_consensus.cc:2229] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:04.791644 7908 raft_consensus.cc:2229] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:04.791682 7908 raft_consensus.cc:2258] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:04.791682 7907 raft_consensus.cc:2258] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:04.791715 7909 tablet_replica.cc:330] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334: stopping tablet replica
I20260620 14:51:04.791786 7909 raft_consensus.cc:2229] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:04.791917 7909 raft_consensus.cc:2258] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:04.791975 7908 ts_tablet_manager.cc:1905] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:04.792243 7909 ts_tablet_manager.cc:1905] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:04.793941 7909 ts_tablet_manager.cc:1918] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:51:04.794018 7909 log.cc:1196] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/wal/wals/442b4c4cd9a54f73b935c504f0b2cb41
I20260620 14:51:04.794269 7909 ts_tablet_manager.cc:1939] T 442b4c4cd9a54f73b935c504f0b2cb41 P 3b12a94c6d7b4afc8375697e988b0334: Deleting consensus metadata
I20260620 14:51:04.794618 7307 catalog_manager.cc:4841] TS 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129:39987): tablet 442b4c4cd9a54f73b935c504f0b2cb41 (table test-table [id=0cb5dfc7ff5c4a3797d65aa3215c81df]) successfully deleted
I20260620 14:51:04.794986 7907 ts_tablet_manager.cc:1905] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:04.796315 7908 ts_tablet_manager.cc:1918] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:51:04.796447 7908 log.cc:1196] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/442b4c4cd9a54f73b935c504f0b2cb41
I20260620 14:51:04.796559 7907 ts_tablet_manager.cc:1918] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:51:04.796644 7907 log.cc:1196] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/442b4c4cd9a54f73b935c504f0b2cb41
I20260620 14:51:04.796777 7908 ts_tablet_manager.cc:1939] T 442b4c4cd9a54f73b935c504f0b2cb41 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting consensus metadata
I20260620 14:51:04.796954 7907 ts_tablet_manager.cc:1939] T 442b4c4cd9a54f73b935c504f0b2cb41 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting consensus metadata
I20260620 14:51:04.797145 7307 catalog_manager.cc:4841] TS 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413): tablet 442b4c4cd9a54f73b935c504f0b2cb41 (table test-table [id=0cb5dfc7ff5c4a3797d65aa3215c81df]) successfully deleted
I20260620 14:51:04.797324 7306 catalog_manager.cc:4841] TS ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130:37965): tablet 442b4c4cd9a54f73b935c504f0b2cb41 (table test-table [id=0cb5dfc7ff5c4a3797d65aa3215c81df]) successfully deleted
Jun 20 14:51:04 dist-test-slave-2m5f krb5kdc[7275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:51:04 dist-test-slave-2m5f krb5kdc[7275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:04.817795 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:04.809171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.5.118.190:38615)
0620 14:51:04.809361 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:04.809363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:04.809375 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:51:04.809487 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:04.809489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:04.809507 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:04.809601 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:04.810323 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:04.811020 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:04.811899 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:04.812144 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:04.812349 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:04.814846 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:04.814848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:04.814848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:04.814855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:04.816560 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:04.817014 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:04.817016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:04.817016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:04.817081 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:04.817330 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:04.817331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:04.817331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:04.817451 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:04.817532 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:04.817646 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:51:04.817720 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":104,"threads_started":1}
I20260620 14:51:04.819578 7319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:04.819770 7319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:04.821411 7457 tablet_service.cc:1467] Processing CreateTablet for tablet c4ccb248f7394c4b840794e9618ad555 (DEFAULT_TABLE table=test-table [id=5e1d67a9df57417889201951f44d1118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:04.821445 7737 tablet_service.cc:1467] Processing CreateTablet for tablet c4ccb248f7394c4b840794e9618ad555 (DEFAULT_TABLE table=test-table [id=5e1d67a9df57417889201951f44d1118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:04.821441 7597 tablet_service.cc:1467] Processing CreateTablet for tablet c4ccb248f7394c4b840794e9618ad555 (DEFAULT_TABLE table=test-table [id=5e1d67a9df57417889201951f44d1118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:04.821557 7737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ccb248f7394c4b840794e9618ad555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:04.821558 7457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ccb248f7394c4b840794e9618ad555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:04.821573 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ccb248f7394c4b840794e9618ad555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:04.822633 7918 tablet_bootstrap.cc:492] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5: Bootstrap starting.
I20260620 14:51:04.823213 7918 tablet_bootstrap.cc:654] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:04.823474 7919 tablet_bootstrap.cc:492] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Bootstrap starting.
I20260620 14:51:04.823797 7918 tablet_bootstrap.cc:492] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5: No bootstrap required, opened a new log
I20260620 14:51:04.823861 7919 tablet_bootstrap.cc:654] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:04.823866 7918 ts_tablet_manager.cc:1397] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:04.823992 7918 raft_consensus.cc:348] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.824086 7918 raft_consensus.cc:374] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:04.824110 7918 raft_consensus.cc:729] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5ba2bd55eb4202af4d804320aa62f5, State: Initialized, Role: FOLLOWER
I20260620 14:51:04.824139 7920 tablet_bootstrap.cc:492] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334: Bootstrap starting.
I20260620 14:51:04.824273 7918 consensus_queue.cc:260] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.824465 7918 ts_tablet_manager.cc:1428] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:04.824564 7920 tablet_bootstrap.cc:654] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:04.825066 7919 tablet_bootstrap.cc:492] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9: No bootstrap required, opened a new log
I20260620 14:51:04.825120 7919 ts_tablet_manager.cc:1397] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:04.825269 7919 raft_consensus.cc:348] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.825349 7919 raft_consensus.cc:374] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:04.825372 7919 raft_consensus.cc:729] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cd25e44ae5452f89cbd4e1ed6cbde9, State: Initialized, Role: FOLLOWER
I20260620 14:51:04.825459 7919 consensus_queue.cc:260] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.825642 7919 ts_tablet_manager.cc:1428] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:04.825871 7920 tablet_bootstrap.cc:492] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334: No bootstrap required, opened a new log
I20260620 14:51:04.825925 7920 ts_tablet_manager.cc:1397] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:04.826053 7920 raft_consensus.cc:348] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.826128 7920 raft_consensus.cc:374] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:04.826159 7920 raft_consensus.cc:729] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b12a94c6d7b4afc8375697e988b0334, State: Initialized, Role: FOLLOWER
I20260620 14:51:04.826215 7920 consensus_queue.cc:260] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.826303 7920 ts_tablet_manager.cc:1428] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:04.852545 7923 raft_consensus.cc:482] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:04.852641 7923 raft_consensus.cc:504] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.852849 7923 leader_election.cc:288] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129:39987), 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413)
I20260620 14:51:04.853067 7477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ccb248f7394c4b840794e9618ad555" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b12a94c6d7b4afc8375697e988b0334" is_pre_election: true
I20260620 14:51:04.853068 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ccb248f7394c4b840794e9618ad555" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" is_pre_election: true
I20260620 14:51:04.853194 7756 raft_consensus.cc:2454] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 0.
I20260620 14:51:04.853193 7477 raft_consensus.cc:2454] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 0.
I20260620 14:51:04.853356 7551 leader_election.cc:302] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79cd25e44ae5452f89cbd4e1ed6cbde9, ec5ba2bd55eb4202af4d804320aa62f5; no voters:
I20260620 14:51:04.853497 7923 raft_consensus.cc:2781] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:04.853541 7923 raft_consensus.cc:482] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:04.853555 7923 raft_consensus.cc:3037] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:04.854270 7923 raft_consensus.cc:504] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.854403 7923 leader_election.cc:288] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 election: Requested vote from peers 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129:39987), 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413)
I20260620 14:51:04.854604 7477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ccb248f7394c4b840794e9618ad555" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b12a94c6d7b4afc8375697e988b0334"
I20260620 14:51:04.854809 7477 raft_consensus.cc:3037] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:04.854954 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ccb248f7394c4b840794e9618ad555" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9"
I20260620 14:51:04.855044 7756 raft_consensus.cc:3037] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:04.856091 7477 raft_consensus.cc:2454] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 1.
I20260620 14:51:04.856316 7551 leader_election.cc:302] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b12a94c6d7b4afc8375697e988b0334, ec5ba2bd55eb4202af4d804320aa62f5; no voters:
I20260620 14:51:04.856380 7756 raft_consensus.cc:2454] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 1.
I20260620 14:51:04.856416 7923 raft_consensus.cc:2781] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:04.856613 7923 raft_consensus.cc:686] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 LEADER]: Becoming Leader. State: Replica: ec5ba2bd55eb4202af4d804320aa62f5, State: Running, Role: LEADER
I20260620 14:51:04.856678 7923 consensus_queue.cc:237] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.857290 7320 catalog_manager.cc:5495] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130). New cstate: current_term: 1 leader_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:04.864573 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:04.861988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.5.118.130:37965)
0620 14:51:04.862126 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:04.862128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:04.862136 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:04.862160 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:04.862161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:04.862177 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:04.862235 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:04.862376 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:04.863058 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:04.863887 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:04.864074 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:04.864129 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:04.864336 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:04.864364 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:04.864406 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:04.864443 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:51:04.864463 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":93,"threads_started":1}
I20260620 14:51:04.866950 7756 raft_consensus.cc:1261] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Refusing update from remote peer ec5ba2bd55eb4202af4d804320aa62f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:04.866962 7477 raft_consensus.cc:1261] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Refusing update from remote peer ec5ba2bd55eb4202af4d804320aa62f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:04.867189 7923 consensus_queue.cc:1035] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:04.867322 7926 consensus_queue.cc:1035] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:04.877211 7320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:04.877368 7320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:04.878039 7320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7: Sending DeleteTablet for 3 replicas of tablet c4ccb248f7394c4b840794e9618ad555
I20260620 14:51:04.878302 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ccb248f7394c4b840794e9618ad555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:04 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:48688
I20260620 14:51:04.878340 7457 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ccb248f7394c4b840794e9618ad555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:04 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:36320
I20260620 14:51:04.878364 7737 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ccb248f7394c4b840794e9618ad555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:04 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37552
I20260620 14:51:04.878408 7907 tablet_replica.cc:330] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5: stopping tablet replica
I20260620 14:51:04.878458 7907 raft_consensus.cc:2229] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:04.878459 7908 tablet_replica.cc:330] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9: stopping tablet replica
I20260620 14:51:04.878512 7907 raft_consensus.cc:2258] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:04.878508 7908 raft_consensus.cc:2229] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:04.878549 7908 raft_consensus.cc:2258] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:04.878643 7909 tablet_replica.cc:330] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334: stopping tablet replica
I20260620 14:51:04.878705 7909 raft_consensus.cc:2229] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:04.878746 7909 raft_consensus.cc:2258] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:04.878945 7907 ts_tablet_manager.cc:1905] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:04.879042 7909 ts_tablet_manager.cc:1905] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:04.880481 7320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:51:04.880594 7908 ts_tablet_manager.cc:1905] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:51:04.880645 7320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:04.880672 7909 ts_tablet_manager.cc:1918] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:04.880733 7909 log.cc:1196] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-0/wal/wals/c4ccb248f7394c4b840794e9618ad555
I20260620 14:51:04.880756 7907 ts_tablet_manager.cc:1918] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:04.880811 7907 log.cc:1196] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/c4ccb248f7394c4b840794e9618ad555
I20260620 14:51:04.880980 7909 ts_tablet_manager.cc:1939] T c4ccb248f7394c4b840794e9618ad555 P 3b12a94c6d7b4afc8375697e988b0334: Deleting consensus metadata
I20260620 14:51:04.881049 7907 ts_tablet_manager.cc:1939] T c4ccb248f7394c4b840794e9618ad555 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting consensus metadata
I20260620 14:51:04.881333 7307 catalog_manager.cc:4841] TS 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129:39987): tablet c4ccb248f7394c4b840794e9618ad555 (table test-table [id=5e1d67a9df57417889201951f44d1118]) successfully deleted
I20260620 14:51:04.881333 7306 catalog_manager.cc:4841] TS ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130:37965): tablet c4ccb248f7394c4b840794e9618ad555 (table test-table [id=5e1d67a9df57417889201951f44d1118]) successfully deleted
I20260620 14:51:04.881834 7597 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7709f7b7824a7f98e22636c3c14626 (DEFAULT_TABLE table=test-table [id=aa4f1272a15644578a6b6c2f715d243f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:04.881837 7457 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7709f7b7824a7f98e22636c3c14626 (DEFAULT_TABLE table=test-table [id=aa4f1272a15644578a6b6c2f715d243f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:04.881945 7597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7709f7b7824a7f98e22636c3c14626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:04.881945 7457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7709f7b7824a7f98e22636c3c14626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:04.882274 7908 ts_tablet_manager.cc:1918] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:04.882316 7908 log.cc:1196] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/c4ccb248f7394c4b840794e9618ad555
I20260620 14:51:04.882479 7908 ts_tablet_manager.cc:1939] T c4ccb248f7394c4b840794e9618ad555 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting consensus metadata
I20260620 14:51:04.883258 7920 tablet_bootstrap.cc:492] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334: Bootstrap starting.
I20260620 14:51:04.883296 7918 tablet_bootstrap.cc:492] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5: Bootstrap starting.
I20260620 14:51:04.883466 7737 tablet_service.cc:1467] Processing CreateTablet for tablet 3f7709f7b7824a7f98e22636c3c14626 (DEFAULT_TABLE table=test-table [id=aa4f1272a15644578a6b6c2f715d243f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:04.883586 7737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f7709f7b7824a7f98e22636c3c14626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:04.883630 7918 tablet_bootstrap.cc:654] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:04.883630 7920 tablet_bootstrap.cc:654] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:04.884289 7918 tablet_bootstrap.cc:492] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5: No bootstrap required, opened a new log
I20260620 14:51:04.884321 7920 tablet_bootstrap.cc:492] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334: No bootstrap required, opened a new log
I20260620 14:51:04.884353 7918 ts_tablet_manager.cc:1397] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:04.884392 7920 ts_tablet_manager.cc:1397] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:04.884474 7918 raft_consensus.cc:348] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.884555 7918 raft_consensus.cc:374] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:04.884588 7918 raft_consensus.cc:729] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5ba2bd55eb4202af4d804320aa62f5, State: Initialized, Role: FOLLOWER
I20260620 14:51:04.884513 7920 raft_consensus.cc:348] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.884617 7920 raft_consensus.cc:374] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:04.884644 7920 raft_consensus.cc:729] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b12a94c6d7b4afc8375697e988b0334, State: Initialized, Role: FOLLOWER
I20260620 14:51:04.884652 7918 consensus_queue.cc:260] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.884698 7920 consensus_queue.cc:260] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.884747 7918 ts_tablet_manager.cc:1428] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:04.884810 7920 ts_tablet_manager.cc:1428] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:04.884938 7919 tablet_bootstrap.cc:492] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Bootstrap starting.
I20260620 14:51:04.885460 7919 tablet_bootstrap.cc:654] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:04.885903 7307 catalog_manager.cc:4841] TS 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413): tablet c4ccb248f7394c4b840794e9618ad555 (table test-table [id=5e1d67a9df57417889201951f44d1118]) successfully deleted
I20260620 14:51:04.886212 7919 tablet_bootstrap.cc:492] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9: No bootstrap required, opened a new log
I20260620 14:51:04.886299 7919 ts_tablet_manager.cc:1397] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:04.886444 7919 raft_consensus.cc:348] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.886502 7919 raft_consensus.cc:374] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:04.886516 7919 raft_consensus.cc:729] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79cd25e44ae5452f89cbd4e1ed6cbde9, State: Initialized, Role: FOLLOWER
I20260620 14:51:04.886574 7919 consensus_queue.cc:260] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:04.886685 7919 ts_tablet_manager.cc:1428] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:05.054365 7933 raft_consensus.cc:482] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:05.054494 7933 raft_consensus.cc:504] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:05.054792 7933 leader_election.cc:288] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129:39987), 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413)
I20260620 14:51:05.055045 7477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7709f7b7824a7f98e22636c3c14626" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b12a94c6d7b4afc8375697e988b0334" is_pre_election: true
I20260620 14:51:05.055153 7477 raft_consensus.cc:2454] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 0.
I20260620 14:51:05.055124 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7709f7b7824a7f98e22636c3c14626" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" is_pre_election: true
I20260620 14:51:05.055249 7756 raft_consensus.cc:2454] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 0.
I20260620 14:51:05.055404 7551 leader_election.cc:302] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b12a94c6d7b4afc8375697e988b0334, ec5ba2bd55eb4202af4d804320aa62f5; no voters:
I20260620 14:51:05.055531 7933 raft_consensus.cc:2781] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:05.055608 7933 raft_consensus.cc:482] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:05.055645 7933 raft_consensus.cc:3037] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:05.056597 7933 raft_consensus.cc:504] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:05.056756 7933 leader_election.cc:288] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 election: Requested vote from peers 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129:39987), 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413)
I20260620 14:51:05.056967 7477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7709f7b7824a7f98e22636c3c14626" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b12a94c6d7b4afc8375697e988b0334"
I20260620 14:51:05.056975 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f7709f7b7824a7f98e22636c3c14626" candidate_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9"
I20260620 14:51:05.057068 7756 raft_consensus.cc:3037] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:05.057070 7477 raft_consensus.cc:3037] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:05.057683 7756 raft_consensus.cc:2454] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 1.
I20260620 14:51:05.057708 7477 raft_consensus.cc:2454] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec5ba2bd55eb4202af4d804320aa62f5 in term 1.
I20260620 14:51:05.057916 7551 leader_election.cc:302] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b12a94c6d7b4afc8375697e988b0334, ec5ba2bd55eb4202af4d804320aa62f5; no voters:
I20260620 14:51:05.058084 7933 raft_consensus.cc:2781] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:05.058158 7933 raft_consensus.cc:686] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 LEADER]: Becoming Leader. State: Replica: ec5ba2bd55eb4202af4d804320aa62f5, State: Running, Role: LEADER
I20260620 14:51:05.058247 7933 consensus_queue.cc:237] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } }
I20260620 14:51:05.058965 7318 catalog_manager.cc:5495] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130). New cstate: current_term: 1 leader_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec5ba2bd55eb4202af4d804320aa62f5" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 37965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:05.075567 7756 raft_consensus.cc:1261] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Refusing update from remote peer ec5ba2bd55eb4202af4d804320aa62f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:05.075572 7477 raft_consensus.cc:1261] T 3f7709f7b7824a7f98e22636c3c14626 P 3b12a94c6d7b4afc8375697e988b0334 [term 1 FOLLOWER]: Refusing update from remote peer ec5ba2bd55eb4202af4d804320aa62f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:05.075853 7933 consensus_queue.cc:1035] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b12a94c6d7b4afc8375697e988b0334" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:05.076004 7923 consensus_queue.cc:1035] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79cd25e44ae5452f89cbd4e1ed6cbde9" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 42413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:51:05.080504 7891 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:51:06.601449 7319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:06.601578 7319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:06.602471 7319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3b0803a8f65433ebc128f788f7b62b7: Sending DeleteTablet for 3 replicas of tablet 3f7709f7b7824a7f98e22636c3c14626
I20260620 14:51:06.602777 7457 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f7709f7b7824a7f98e22636c3c14626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:06 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:36320
I20260620 14:51:06.602813 7737 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f7709f7b7824a7f98e22636c3c14626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:06 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37552
I20260620 14:51:06.602814 7597 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f7709f7b7824a7f98e22636c3c14626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:06 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:48688
I20260620 14:51:06.602949 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 7392
I20260620 14:51:06.603703 7964 tablet_replica.cc:330] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5: stopping tablet replica
I20260620 14:51:06.603760 7964 raft_consensus.cc:2229] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:06.603864 7964 raft_consensus.cc:2258] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:06.603986 7965 tablet_replica.cc:330] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9: stopping tablet replica
I20260620 14:51:06.604043 7965 raft_consensus.cc:2229] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:06.604090 7965 raft_consensus.cc:2258] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:06.604334 7964 ts_tablet_manager.cc:1905] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:06.605858 7965 ts_tablet_manager.cc:1905] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:06.606055 7964 ts_tablet_manager.cc:1918] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:51:06.606117 7964 log.cc:1196] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/3f7709f7b7824a7f98e22636c3c14626
I20260620 14:51:06.606474 7964 ts_tablet_manager.cc:1939] T 3f7709f7b7824a7f98e22636c3c14626 P ec5ba2bd55eb4202af4d804320aa62f5: Deleting consensus metadata
I20260620 14:51:06.606845 7306 catalog_manager.cc:4841] TS ec5ba2bd55eb4202af4d804320aa62f5 (127.5.118.130:37965): tablet 3f7709f7b7824a7f98e22636c3c14626 (table test-table [id=aa4f1272a15644578a6b6c2f715d243f]) successfully deleted
I20260620 14:51:06.607438 7965 ts_tablet_manager.cc:1918] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:51:06.607491 7965 log.cc:1196] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/3f7709f7b7824a7f98e22636c3c14626
I20260620 14:51:06.607678 7965 ts_tablet_manager.cc:1939] T 3f7709f7b7824a7f98e22636c3c14626 P 79cd25e44ae5452f89cbd4e1ed6cbde9: Deleting consensus metadata
I20260620 14:51:06.608198 7307 catalog_manager.cc:4841] TS 79cd25e44ae5452f89cbd4e1ed6cbde9 (127.5.118.131:42413): tablet 3f7709f7b7824a7f98e22636c3c14626 (table test-table [id=aa4f1272a15644578a6b6c2f715d243f]) successfully deleted
W20260620 14:51:06.610532 7307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:39987 (error 108)
I20260620 14:51:06.610823 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 7532
I20260620 14:51:06.616670 7967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.616376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39987 (local address 127.0.0.1:36354)
0620 14:51:06.616541 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.616609 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
W20260620 14:51:06.616847 7307 catalog_manager.cc:4584] TS 3b12a94c6d7b4afc8375697e988b0334 (127.5.118.129:39987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f7709f7b7824a7f98e22636c3c14626: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39987: connect: Connection refused (error 111)
I20260620 14:51:06.617183 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 7672
I20260620 14:51:06.622822 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 7288
2026-06-20T14:51:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6089 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:06 dist-test-slave-2m5f krb5kdc[7970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:06 dist-test-slave-2m5f krb5kdc[7970](info): set up 2 sockets
Jun 20 14:51:06 dist-test-slave-2m5f krb5kdc[7970](info): commencing operation
krb5kdc: starting...
W20260620 14:51:07.768594 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:07 dist-test-slave-2m5f krb5kdc[7970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:51:07.895663 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:34149
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:34149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:07.978340 7983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:07.978606 7983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:07.978645 7983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:07.979835 7983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:07.979915 7983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:07.979939 7983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:07.979961 7983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:07.979991 7983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:07.981631 7983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:34149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:34149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:07.981925 7983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:07.982167 7983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:07.987078 7989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:07.987345 7983 server_base.cc:1034] running on GCE node
W20260620 14:51:07.987087 7991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:07.987063 7988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:07.987695 7983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:07.987958 7983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:07.989110 7983 hybrid_clock.cc:648] HybridClock initialized: now 1781967067989039 us; error 86 us; skew 500 ppm
Jun 20 14:51:07 dist-test-slave-2m5f krb5kdc[7970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967067, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:07.991902 7983 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:51:07.992425 7983 webserver.cc:460] Webserver started at http://127.5.118.190:43115/ using document root <none> and password file <none>
I20260620 14:51:07.992677 7983 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:07.992727 7983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:07.992926 7983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:07.994138 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "01e53e6249e542f4b89feef180f98943"
format_stamp: "Formatted at 2026-06-20 14:51:07 on dist-test-slave-2m5f"
server_key: "587b98f97e5dee23fb1fdcd4e36242e5"
server_key_iv: "3e6918f30d0a27a9ecde950c8b32b933"
server_key_version: "encryptionkey@0"
I20260620 14:51:07.994549 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "01e53e6249e542f4b89feef180f98943"
format_stamp: "Formatted at 2026-06-20 14:51:07 on dist-test-slave-2m5f"
server_key: "587b98f97e5dee23fb1fdcd4e36242e5"
server_key_iv: "3e6918f30d0a27a9ecde950c8b32b933"
server_key_version: "encryptionkey@0"
I20260620 14:51:07.996042 7983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:07.997081 7998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:07.997365 7983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:07.997483 7983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "01e53e6249e542f4b89feef180f98943"
format_stamp: "Formatted at 2026-06-20 14:51:07 on dist-test-slave-2m5f"
server_key: "587b98f97e5dee23fb1fdcd4e36242e5"
server_key_iv: "3e6918f30d0a27a9ecde950c8b32b933"
server_key_version: "encryptionkey@0"
I20260620 14:51:07.997571 7983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:08.021857 7983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:08.024423 7983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:08.024610 7983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:08.028968 7983 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:34149
I20260620 14:51:08.029048 8050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:34149 every 8 connection(s)
I20260620 14:51:08.029458 7983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:08.030443 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:08.031121 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 7983
I20260620 14:51:08.031222 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:08.031414 5594 external_mini_cluster.cc:1428] Setting key 7251b2d35477c409d135f6fec94868cf
I20260620 14:51:08.033857 8051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943: Bootstrap starting.
I20260620 14:51:08.034637 8051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:08.034973 8051 log.cc:824] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:08.035770 8051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943: No bootstrap required, opened a new log
I20260620 14:51:08.038455 8051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e53e6249e542f4b89feef180f98943" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34149 } }
I20260620 14:51:08.038606 8051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:08.038630 8051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e53e6249e542f4b89feef180f98943, State: Initialized, Role: FOLLOWER
I20260620 14:51:08.038793 8051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e53e6249e542f4b89feef180f98943" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34149 } }
I20260620 14:51:08.038892 8051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:08.038941 8051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:08.038996 8051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:51:08 dist-test-slave-2m5f krb5kdc[7970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:08.039654 8051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e53e6249e542f4b89feef180f98943" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34149 } }
I20260620 14:51:08.039753 8051 leader_election.cc:302] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01e53e6249e542f4b89feef180f98943; no voters:
I20260620 14:51:08.039968 8051 leader_election.cc:288] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:08.040006 8056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:08.040200 8056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [term 1 LEADER]: Becoming Leader. State: Replica: 01e53e6249e542f4b89feef180f98943, State: Running, Role: LEADER
I20260620 14:51:08.040337 8051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:08.040357 8056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e53e6249e542f4b89feef180f98943" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34149 } }
I20260620 14:51:08.040879 8057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01e53e6249e542f4b89feef180f98943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e53e6249e542f4b89feef180f98943" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34149 } } }
I20260620 14:51:08.041143 8057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:08.041067 8058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01e53e6249e542f4b89feef180f98943. Latest consensus state: current_term: 1 leader_uuid: "01e53e6249e542f4b89feef180f98943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e53e6249e542f4b89feef180f98943" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34149 } } }
I20260620 14:51:08.042222 8058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943 [sys.catalog]: This master's current role is: LEADER
W20260620 14:51:08.042397 8071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:51:08.042593 8071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:51:08.042672 8065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:08.042989 8065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:08.044695 8065 catalog_manager.cc:1347] Generated new cluster ID: dab9815fa45145a8bec7eef35173a8ca
I20260620 14:51:08.044762 8065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:08.045153 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:08.032047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.5.118.190:34149)
0620 14:51:08.032280 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:08.032293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:08.032332 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:51:08.032938 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:08.032948 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:08.033015 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:08.033196 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:08.033860 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.035454 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.036389 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.037068 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.037208 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:08.040040 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:08.040045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:08.040046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:08.040061 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:08.043230 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:08.043676 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:08.043679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:08.043679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:08.043788 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:08.044025 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:08.044026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:08.044027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:08.044404 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:08.044589 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:08.044695 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 14:51:08.044973 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
I20260620 14:51:08.060598 8065 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:08.061424 8065 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:08.066893 8065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01e53e6249e542f4b89feef180f98943: Generated new TSK 0
I20260620 14:51:08.067108 8065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:51:08.095570 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34149
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:08.184725 8077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:08.184944 8077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:08.184970 8077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:08.186275 8077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:08.186329 8077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:08.186360 8077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:08.187996 8077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:08.188289 8077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:08.188544 8077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:08.194550 8083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:08.194545 8085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:08.194725 8077 server_base.cc:1034] running on GCE node
W20260620 14:51:08.194550 8082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:08.195152 8077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:08.195417 8077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:08.196583 8077 hybrid_clock.cc:648] HybridClock initialized: now 1781967068196559 us; error 40 us; skew 500 ppm
Jun 20 14:51:08 dist-test-slave-2m5f krb5kdc[7970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967068, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:08.199040 8077 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:51:08.199559 8077 webserver.cc:460] Webserver started at http://127.5.118.129:34643/ using document root <none> and password file <none>
I20260620 14:51:08.199829 8077 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:08.199906 8077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:08.200074 8077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:08.201436 8077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "11bd0b55aa4848c98ddaaba289063620"
format_stamp: "Formatted at 2026-06-20 14:51:08 on dist-test-slave-2m5f"
server_key: "e74ce1ef46d0666697bf69ccf47babc0"
server_key_iv: "2f6cd5d1de2887a16d2f7970c7d914fd"
server_key_version: "encryptionkey@0"
I20260620 14:51:08.201841 8077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "11bd0b55aa4848c98ddaaba289063620"
format_stamp: "Formatted at 2026-06-20 14:51:08 on dist-test-slave-2m5f"
server_key: "e74ce1ef46d0666697bf69ccf47babc0"
server_key_iv: "2f6cd5d1de2887a16d2f7970c7d914fd"
server_key_version: "encryptionkey@0"
I20260620 14:51:08.203356 8077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:08.204295 8092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:08.204504 8077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:08.204615 8077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "11bd0b55aa4848c98ddaaba289063620"
format_stamp: "Formatted at 2026-06-20 14:51:08 on dist-test-slave-2m5f"
server_key: "e74ce1ef46d0666697bf69ccf47babc0"
server_key_iv: "2f6cd5d1de2887a16d2f7970c7d914fd"
server_key_version: "encryptionkey@0"
I20260620 14:51:08.204715 8077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:08.230331 8077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:08.232885 8077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:08.233079 8077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:08.233345 8077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:08.233769 8077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:08.233836 8077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:08.233877 8077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:08.233932 8077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:08.240509 8077 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:32801
I20260620 14:51:08.240549 8205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:32801 every 8 connection(s)
I20260620 14:51:08.240926 8077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:51:08 dist-test-slave-2m5f krb5kdc[7970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967068, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:08.251058 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 8077
I20260620 14:51:08.251171 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:08.251375 5594 external_mini_cluster.cc:1428] Setting key cd66cbc56cfa4c4cbd9543e6de5181ea
I20260620 14:51:08.251457 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:08.241712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:39315 (local address 127.5.118.190:34149)
0620 14:51:08.241797 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:08.241799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:08.242083 (+ 284us) server_negotiation.cc:407] Connection header received
0620 14:51:08.242671 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:08.242672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:08.242685 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:08.242738 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:08.243743 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.244334 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.245359 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.245582 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.245668 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:08.248222 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:08.248224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:08.248224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:08.248235 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:08.249657 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:08.250243 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:08.250244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:08.250244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:08.250329 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:08.250697 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:08.250698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:08.250698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:08.250804 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:08.250854 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:08.251254 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 14:51:08.251357 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:51:08.251690 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:08.241714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:34149 (local address 127.5.118.129:39315)
0620 14:51:08.241905 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:08.241967 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:08.242556 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:08.242825 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:08.242829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:08.243435 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:08.243634 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:08.243639 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.244436 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.244437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:08.245265 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:08.245268 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.245679 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.245680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:08.245819 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:08.246374 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:08.246382 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:08.248076 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:08.249739 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:08.249913 (+ 174us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:08.249915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:08.250172 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:08.250421 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:08.250422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:08.250422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:08.250549 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:08.250890 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:08.250897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:08.251036 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 14:51:08.251190 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 14:51:08.251376 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260620 14:51:08.252329 8206 heartbeater.cc:347] Connected to a master server at 127.5.118.190:34149
I20260620 14:51:08.252429 8206 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:08.252633 8206 heartbeater.cc:511] Master 127.5.118.190:34149 requested a full tablet report, sending...
I20260620 14:51:08.253232 8015 ts_manager.cc:194] Registered new tserver with Master: 11bd0b55aa4848c98ddaaba289063620 (127.5.118.129:32801)
I20260620 14:51:08.254150 8015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:39315
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:51:08.279451 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34149
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:08.362809 8211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:08.363019 8211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:08.363045 8211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:08.364579 8211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:08.364655 8211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:08.364699 8211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:08.366408 8211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:08.366670 8211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:08.366925 8211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:08.371922 8217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:08.371922 8216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:08.371954 8219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:08.372220 8211 server_base.cc:1034] running on GCE node
I20260620 14:51:08.372423 8211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:08.372740 8211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:08.373899 8211 hybrid_clock.cc:648] HybridClock initialized: now 1781967068373874 us; error 37 us; skew 500 ppm
Jun 20 14:51:08 dist-test-slave-2m5f krb5kdc[7970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967068, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:08.376672 8211 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:51:08.377099 8211 webserver.cc:460] Webserver started at http://127.5.118.130:43859/ using document root <none> and password file <none>
I20260620 14:51:08.377372 8211 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:08.377418 8211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:08.377588 8211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:08.378654 8211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "53ac13c3620e4e2596fa1f325110f191"
format_stamp: "Formatted at 2026-06-20 14:51:08 on dist-test-slave-2m5f"
server_key: "d82451a1428bc513fdbf3d6bf6250615"
server_key_iv: "20467b6dcd654c138551a661b5fb6a17"
server_key_version: "encryptionkey@0"
I20260620 14:51:08.379026 8211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "53ac13c3620e4e2596fa1f325110f191"
format_stamp: "Formatted at 2026-06-20 14:51:08 on dist-test-slave-2m5f"
server_key: "d82451a1428bc513fdbf3d6bf6250615"
server_key_iv: "20467b6dcd654c138551a661b5fb6a17"
server_key_version: "encryptionkey@0"
I20260620 14:51:08.380471 8211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:51:08.381318 8226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:08.381508 8211 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:08.381585 8211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "53ac13c3620e4e2596fa1f325110f191"
format_stamp: "Formatted at 2026-06-20 14:51:08 on dist-test-slave-2m5f"
server_key: "d82451a1428bc513fdbf3d6bf6250615"
server_key_iv: "20467b6dcd654c138551a661b5fb6a17"
server_key_version: "encryptionkey@0"
I20260620 14:51:08.381650 8211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:08.393164 8211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:08.395151 8211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:08.395264 8211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:08.395459 8211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:08.395787 8211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:08.395821 8211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:08.395844 8211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:08.395915 8211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:08.402063 8211 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:46227
I20260620 14:51:08.402146 8339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:46227 every 8 connection(s)
I20260620 14:51:08.402477 8211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:08.404759 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 8211
I20260620 14:51:08.404865 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:08.405045 5594 external_mini_cluster.cc:1428] Setting key f20e7b8b68a1ef39d7951741dc0f2c3f
Jun 20 14:51:08 dist-test-slave-2m5f krb5kdc[7970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967068, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:08.413866 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:08.403230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:58795 (local address 127.5.118.190:34149)
0620 14:51:08.403307 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:08.403309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:08.403511 (+ 202us) server_negotiation.cc:407] Connection header received
0620 14:51:08.404007 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:08.404008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:08.404019 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:08.404081 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:08.405355 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.406087 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.407622 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.407998 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.408086 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:08.410567 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:08.410568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:08.410569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:08.410577 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:08.412149 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:08.412745 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:08.412746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:08.412746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:08.412821 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:08.413166 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:08.413167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:08.413167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:08.413281 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:08.413337 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:08.413728 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 14:51:08.413799 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:51:08.414139 8342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:08.403198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:34149 (local address 127.5.118.130:58795)
0620 14:51:08.403385 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:08.403423 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:08.403919 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:08.404138 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:08.404142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:08.404929 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:08.405251 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:08.405259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.406189 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.406190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:08.407523 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:08.407526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.408087 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.408088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:08.408243 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:08.408795 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:08.408804 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:08.410462 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:08.412309 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:08.412317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:08.412320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:08.412647 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:08.412904 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:08.412905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:08.412905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:08.413062 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:08.413355 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:08.413362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:08.413547 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 14:51:08.413685 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 14:51:08.413879 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
I20260620 14:51:08.414737 8340 heartbeater.cc:347] Connected to a master server at 127.5.118.190:34149
I20260620 14:51:08.414862 8340 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:08.415043 8340 heartbeater.cc:511] Master 127.5.118.190:34149 requested a full tablet report, sending...
I20260620 14:51:08.415536 8015 ts_manager.cc:194] Registered new tserver with Master: 53ac13c3620e4e2596fa1f325110f191 (127.5.118.130:46227)
I20260620 14:51:08.415961 8015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:58795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:51:08.432963 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34149
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:32775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:08.516901 8345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:08.517150 8345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:08.517191 8345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:08.518386 8345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:08.518457 8345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:08.518502 8345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:08.519966 8345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:32775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:08.520267 8345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:08.520515 8345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:08.525538 8351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:08.525707 8345 server_base.cc:1034] running on GCE node
W20260620 14:51:08.525802 8350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:08.525575 8353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:08.526069 8345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:08.526322 8345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:08.527485 8345 hybrid_clock.cc:648] HybridClock initialized: now 1781967068527487 us; error 43 us; skew 500 ppm
Jun 20 14:51:08 dist-test-slave-2m5f krb5kdc[7970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967068, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:08.530002 8345 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:51:08.530475 8345 webserver.cc:460] Webserver started at http://127.5.118.131:33115/ using document root <none> and password file <none>
I20260620 14:51:08.530723 8345 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:08.530802 8345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:08.530948 8345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:08.532014 8345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "5553eded0d9d41cba8138320ac03bd19"
format_stamp: "Formatted at 2026-06-20 14:51:08 on dist-test-slave-2m5f"
server_key: "897573e85b53d8e770bd717a390c161a"
server_key_iv: "6bba1bd933a0fa8b5d9ab5e664ea9b35"
server_key_version: "encryptionkey@0"
I20260620 14:51:08.532398 8345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "5553eded0d9d41cba8138320ac03bd19"
format_stamp: "Formatted at 2026-06-20 14:51:08 on dist-test-slave-2m5f"
server_key: "897573e85b53d8e770bd717a390c161a"
server_key_iv: "6bba1bd933a0fa8b5d9ab5e664ea9b35"
server_key_version: "encryptionkey@0"
I20260620 14:51:08.533802 8345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:08.534559 8360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:08.534840 8345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:08.534943 8345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "5553eded0d9d41cba8138320ac03bd19"
format_stamp: "Formatted at 2026-06-20 14:51:08 on dist-test-slave-2m5f"
server_key: "897573e85b53d8e770bd717a390c161a"
server_key_iv: "6bba1bd933a0fa8b5d9ab5e664ea9b35"
server_key_version: "encryptionkey@0"
I20260620 14:51:08.535020 8345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:08.560954 8345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:08.563242 8345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:08.563410 8345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:08.563654 8345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:08.563982 8345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:08.564035 8345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:08.564076 8345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:08.564105 8345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:08.570429 8345 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:44563
I20260620 14:51:08.570472 8473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:44563 every 8 connection(s)
I20260620 14:51:08.570817 8345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:51:08 dist-test-slave-2m5f krb5kdc[7970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967068, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:08.578574 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 8345
I20260620 14:51:08.578682 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:08.578938 5594 external_mini_cluster.cc:1428] Setting key a35f59c27179f2cd5a975b5013263c30
I20260620 14:51:08.580912 8054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:08.571558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:52121 (local address 127.5.118.190:34149)
0620 14:51:08.571644 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:08.571647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:08.571867 (+ 220us) server_negotiation.cc:407] Connection header received
0620 14:51:08.572438 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:08.572439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:08.572451 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:08.572504 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:08.573428 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.574082 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.575146 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.575381 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.575454 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:08.577648 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:08.577649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:08.577649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:08.577657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:08.579279 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:08.579823 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:08.579824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:08.579824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:08.579896 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:08.580239 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:08.580240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:08.580240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:08.580380 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:08.580441 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:08.580772 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 14:51:08.580845 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:51:08.581020 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:08.571528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:34149 (local address 127.5.118.131:52121)
0620 14:51:08.571714 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:08.571775 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:08.572299 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:08.572567 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:08.572570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:08.573130 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:08.573325 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:08.573330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.574208 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.574209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:08.575040 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:08.575043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:08.575467 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:08.575468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:08.575571 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:08.576085 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:08.576093 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:08.577556 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:08.579404 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:08.579412 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:08.579415 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:08.579688 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:08.579980 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:08.579981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:08.579982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:08.580118 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:08.580457 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:08.580462 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:08.580588 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 14:51:08.580683 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:51:08.580811 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":113,"threads_started":1}
I20260620 14:51:08.581499 8474 heartbeater.cc:347] Connected to a master server at 127.5.118.190:34149
I20260620 14:51:08.581585 8474 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:08.581806 8474 heartbeater.cc:511] Master 127.5.118.190:34149 requested a full tablet report, sending...
I20260620 14:51:08.582160 8015 ts_manager.cc:194] Registered new tserver with Master: 5553eded0d9d41cba8138320ac03bd19 (127.5.118.131:44563)
I20260620 14:51:08.582602 8015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:52121
I20260620 14:51:08.591313 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:51:08.603650 8054 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:08.602621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.5.118.190:34149)
0620 14:51:08.602753 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:08.602757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:08.602892 (+ 135us) server_negotiation.cc:407] Connection header received
0620 14:51:08.602935 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:08.602936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:08.602947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:08.603001 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:08.603575 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50238: BlockingRecv error: recv got EOF from 127.0.0.1:50238 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 14:51:08.604651 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 8077
I20260620 14:51:08.609259 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 8211
I20260620 14:51:08.613626 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 8345
I20260620 14:51:08.617784 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 7983
2026-06-20T14:51:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1993 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:08 dist-test-slave-2m5f krb5kdc[8486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:08 dist-test-slave-2m5f krb5kdc[8486](info): set up 2 sockets
Jun 20 14:51:08 dist-test-slave-2m5f krb5kdc[8486](info): commencing operation
krb5kdc: starting...
W20260620 14:51:09.684626 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:09 dist-test-slave-2m5f krb5kdc[8486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:51:09.811156 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:36919
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:37737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:36919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:09.894248 8499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:09.894582 8499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:09.894627 8499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:09.895856 8499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:09.895924 8499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:09.895947 8499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:09.895970 8499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:09.895998 8499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:09.897543 8499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:37737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:36919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:36919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:09.897840 8499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:09.898080 8499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:09.903251 8504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:09.903239 8505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:09.903294 8499 server_base.cc:1034] running on GCE node
W20260620 14:51:09.903235 8507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:09.903733 8499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:09.904055 8499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:09.905265 8499 hybrid_clock.cc:648] HybridClock initialized: now 1781967069905263 us; error 60 us; skew 500 ppm
Jun 20 14:51:09 dist-test-slave-2m5f krb5kdc[8486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967069, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:09.907938 8499 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:51:09.908491 8499 webserver.cc:460] Webserver started at http://127.5.118.190:45937/ using document root <none> and password file <none>
I20260620 14:51:09.908736 8499 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:09.908805 8499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:09.908969 8499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:09.910058 8499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "638ed762d358443995f94e53b444a88c"
format_stamp: "Formatted at 2026-06-20 14:51:09 on dist-test-slave-2m5f"
server_key: "c10528f3705d75b961742bf0c82473cd"
server_key_iv: "f2b5195a284d5b2f8e6f4532e40fa755"
server_key_version: "encryptionkey@0"
I20260620 14:51:09.910445 8499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "638ed762d358443995f94e53b444a88c"
format_stamp: "Formatted at 2026-06-20 14:51:09 on dist-test-slave-2m5f"
server_key: "c10528f3705d75b961742bf0c82473cd"
server_key_iv: "f2b5195a284d5b2f8e6f4532e40fa755"
server_key_version: "encryptionkey@0"
I20260620 14:51:09.911861 8499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:51:09.912787 8514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:09.912971 8499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:51:09.913075 8499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "638ed762d358443995f94e53b444a88c"
format_stamp: "Formatted at 2026-06-20 14:51:09 on dist-test-slave-2m5f"
server_key: "c10528f3705d75b961742bf0c82473cd"
server_key_iv: "f2b5195a284d5b2f8e6f4532e40fa755"
server_key_version: "encryptionkey@0"
I20260620 14:51:09.913209 8499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:09.939620 8499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:09.941826 8499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:09.942020 8499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:09.946499 8499 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:36919
I20260620 14:51:09.946537 8566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:36919 every 8 connection(s)
I20260620 14:51:09.947029 8499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:09.948033 8567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:09.950435 8567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c: Bootstrap starting.
I20260620 14:51:09.951094 8567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:09.951406 8567 log.cc:824] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:09.952211 8567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c: No bootstrap required, opened a new log
I20260620 14:51:09.953830 8567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638ed762d358443995f94e53b444a88c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36919 } }
I20260620 14:51:09.953976 8567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:09.953998 8567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638ed762d358443995f94e53b444a88c, State: Initialized, Role: FOLLOWER
I20260620 14:51:09.954154 8567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638ed762d358443995f94e53b444a88c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36919 } }
I20260620 14:51:09.954231 8567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:09.954257 8567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:09.954316 8567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:09.955180 8567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638ed762d358443995f94e53b444a88c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36919 } }
I20260620 14:51:09.955276 8567 leader_election.cc:302] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 638ed762d358443995f94e53b444a88c; no voters:
I20260620 14:51:09.955459 8567 leader_election.cc:288] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:09.955590 8570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:09.955765 8567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:09.955855 8570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [term 1 LEADER]: Becoming Leader. State: Replica: 638ed762d358443995f94e53b444a88c, State: Running, Role: LEADER
I20260620 14:51:09.955991 8570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638ed762d358443995f94e53b444a88c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36919 } }
I20260620 14:51:09.956413 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 8499
I20260620 14:51:09.956511 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:09.956547 8572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 638ed762d358443995f94e53b444a88c. Latest consensus state: current_term: 1 leader_uuid: "638ed762d358443995f94e53b444a88c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638ed762d358443995f94e53b444a88c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36919 } } }
I20260620 14:51:09.956631 8572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:09.956730 5594 external_mini_cluster.cc:1428] Setting key eb2f02d95a775f934b5e01dae20e59e7
I20260620 14:51:09.956748 8571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "638ed762d358443995f94e53b444a88c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638ed762d358443995f94e53b444a88c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36919 } } }
I20260620 14:51:09.956807 8571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:09.956892 8579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:09.957394 8579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:09.959262 8579 catalog_manager.cc:1347] Generated new cluster ID: 0eb2dd1f3e3942a9ba8170856aa4bee6
I20260620 14:51:09.959318 8579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:09.964715 8579 catalog_manager.cc:1370] Generated new certificate authority record
Jun 20 14:51:09 dist-test-slave-2m5f krb5kdc[8486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:09.965461 8579 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:09.969398 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:09.957353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.5.118.190:36919)
0620 14:51:09.957537 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:09.957547 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:09.957588 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:51:09.957821 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:09.957829 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:09.957893 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:09.958011 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:09.959022 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:09.960472 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:09.961623 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:09.962313 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:09.962399 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:09.965462 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:09.965469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:09.965469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:09.965486 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:09.967645 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:09.968084 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:09.968087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:09.968088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:09.968243 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:09.968450 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:09.968450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:09.968451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:09.968757 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:09.968916 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:09.968990 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:51:09.969152 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260620 14:51:09.970319 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:51:09.972787 8579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 638ed762d358443995f94e53b444a88c: Generated new TSK 0
I20260620 14:51:09.972980 8579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:51:09.985409 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:09.981224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56358 (local address 127.5.118.190:36919)
0620 14:51:09.981290 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:09.981292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:09.981400 (+ 108us) server_negotiation.cc:407] Connection header received
0620 14:51:09.981449 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:09.981450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:09.981463 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:09.981538 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:09.981913 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:09.982782 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:09.983595 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:09.983946 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:09.984021 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:09.984858 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:09.984859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:09.984860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:09.985103 (+ 243us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:09.985124 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:09.985131 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:09.985196 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:09.985291 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:51:09.985324 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:51:09.988248 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 8499
2026-06-20T14:51:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1368 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:10 dist-test-slave-2m5f krb5kdc[8600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:10 dist-test-slave-2m5f krb5kdc[8600](info): set up 2 sockets
Jun 20 14:51:10 dist-test-slave-2m5f krb5kdc[8600](info): commencing operation
krb5kdc: starting...
W20260620 14:51:11.033102 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:11 dist-test-slave-2m5f krb5kdc[8600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
W20260620 14:51:11.379540 5594 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.118.190: real 0.247s user 0.000s sys 0.002s
I20260620 14:51:11.379982 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42153
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:42153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:11.463388 8613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:11.463639 8613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:11.463663 8613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:11.465054 8613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:11.465107 8613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:11.465122 8613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:11.465137 8613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:11.465148 8613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:11.466775 8613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:42153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:11.467073 8613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:11.467329 8613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:11.472558 8618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:11.472558 8621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:11.472587 8619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:11.472819 8613 server_base.cc:1034] running on GCE node
I20260620 14:51:11.473094 8613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:11.473438 8613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:11.474629 8613 hybrid_clock.cc:648] HybridClock initialized: now 1781967071474620 us; error 53 us; skew 500 ppm
Jun 20 14:51:11 dist-test-slave-2m5f krb5kdc[8600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967071, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:11.477713 8613 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:51:11.478327 8613 webserver.cc:460] Webserver started at http://127.5.118.190:39911/ using document root <none> and password file <none>
I20260620 14:51:11.478608 8613 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:11.478688 8613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:11.478861 8613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:11.480129 8613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "e1cdec72cc2d4f17bdc1b0c7524de7ac"
format_stamp: "Formatted at 2026-06-20 14:51:11 on dist-test-slave-2m5f"
server_key: "f32249932300aafd2aad889bdd46a0b3"
server_key_iv: "26579b02c238de6a1a4ed73e03ab9854"
server_key_version: "encryptionkey@0"
I20260620 14:51:11.480609 8613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "e1cdec72cc2d4f17bdc1b0c7524de7ac"
format_stamp: "Formatted at 2026-06-20 14:51:11 on dist-test-slave-2m5f"
server_key: "f32249932300aafd2aad889bdd46a0b3"
server_key_iv: "26579b02c238de6a1a4ed73e03ab9854"
server_key_version: "encryptionkey@0"
I20260620 14:51:11.482277 8613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:11.483201 8628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:11.483451 8613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:11.483564 8613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "e1cdec72cc2d4f17bdc1b0c7524de7ac"
format_stamp: "Formatted at 2026-06-20 14:51:11 on dist-test-slave-2m5f"
server_key: "f32249932300aafd2aad889bdd46a0b3"
server_key_iv: "26579b02c238de6a1a4ed73e03ab9854"
server_key_version: "encryptionkey@0"
I20260620 14:51:11.483661 8613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:11.515237 8613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:11.517762 8613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:11.517971 8613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:11.522640 8613 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:42153
I20260620 14:51:11.522766 8680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:42153 every 8 connection(s)
I20260620 14:51:11.523137 8613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:11.524333 8681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:11.525702 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 8613
I20260620 14:51:11.525815 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:11.525981 5594 external_mini_cluster.cc:1428] Setting key d90863b9092a80d70087a2b1f76c8a99
I20260620 14:51:11.527315 8681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac: Bootstrap starting.
I20260620 14:51:11.528064 8681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:11.528414 8681 log.cc:824] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:11.529304 8681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac: No bootstrap required, opened a new log
I20260620 14:51:11.531478 8681 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdec72cc2d4f17bdc1b0c7524de7ac" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:51:11.531668 8681 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:11.531741 8681 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cdec72cc2d4f17bdc1b0c7524de7ac, State: Initialized, Role: FOLLOWER
I20260620 14:51:11.531927 8681 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdec72cc2d4f17bdc1b0c7524de7ac" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:51:11.532047 8681 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:11.532099 8681 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:11.532152 8681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:51:11 dist-test-slave-2m5f krb5kdc[8600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:11.538015 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:11.526719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.5.118.190:42153)
0620 14:51:11.526944 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:11.526955 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:11.526990 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:51:11.527525 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:11.527534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:11.527600 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:11.527786 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:11.528545 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.529724 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.530529 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:11.531000 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:11.531136 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:11.534627 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:11.534632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:11.534632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:11.534647 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:11.536401 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:11.536792 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:11.536799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:11.536799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:11.536889 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:11.537096 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:11.537097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:11.537097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:11.537361 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:11.537509 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:11.537636 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:51:11.537786 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":109,"threads_started":1}
I20260620 14:51:11.566262 8681 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdec72cc2d4f17bdc1b0c7524de7ac" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:51:11.566434 8681 leader_election.cc:302] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1cdec72cc2d4f17bdc1b0c7524de7ac; no voters:
I20260620 14:51:11.566805 8681 leader_election.cc:288] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:11.566864 8686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:11.567075 8686 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [term 1 LEADER]: Becoming Leader. State: Replica: e1cdec72cc2d4f17bdc1b0c7524de7ac, State: Running, Role: LEADER
I20260620 14:51:11.567157 8681 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:11.567256 8686 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdec72cc2d4f17bdc1b0c7524de7ac" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:51:11.567711 8687 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1cdec72cc2d4f17bdc1b0c7524de7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdec72cc2d4f17bdc1b0c7524de7ac" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } } }
I20260620 14:51:11.567806 8687 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:11.568023 8688 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1cdec72cc2d4f17bdc1b0c7524de7ac. Latest consensus state: current_term: 1 leader_uuid: "e1cdec72cc2d4f17bdc1b0c7524de7ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1cdec72cc2d4f17bdc1b0c7524de7ac" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } } }
I20260620 14:51:11.568111 8688 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:11.568295 8693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:11.568849 8693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:11.571408 8693 catalog_manager.cc:1347] Generated new cluster ID: 3b1f54059f3d49758740bbe0b5ea63a8
I20260620 14:51:11.571480 8693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:11.606870 8693 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:11.607498 8693 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:11.611485 8693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1cdec72cc2d4f17bdc1b0c7524de7ac: Generated new TSK 0
I20260620 14:51:11.611711 8693 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
W20260620 14:51:12.735651 5594 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.118.129: real 1.111s user 0.000s sys 0.003s
I20260620 14:51:12.736063 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:12.823432 8707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:12.823729 8707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:12.823774 8707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:12.825042 8707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:12.825110 8707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:12.825155 8707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:12.826786 8707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:12.827066 8707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:12.827319 8707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:12.832891 8715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:12.832895 8713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:12.832901 8712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:12.833204 8707 server_base.cc:1034] running on GCE node
I20260620 14:51:12.833447 8707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:12.833834 8707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:12.835028 8707 hybrid_clock.cc:648] HybridClock initialized: now 1781967072835015 us; error 28 us; skew 500 ppm
Jun 20 14:51:12 dist-test-slave-2m5f krb5kdc[8600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967072, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:12.842002 8707 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:51:12.842550 8707 webserver.cc:460] Webserver started at http://127.5.118.129:37313/ using document root <none> and password file <none>
I20260620 14:51:12.842823 8707 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:12.842893 8707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:12.843055 8707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:12.844291 8707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "5af9008904e54b6b88c7acedc852babd"
format_stamp: "Formatted at 2026-06-20 14:51:12 on dist-test-slave-2m5f"
server_key: "7afa8984ef1a1c1e6e2a7ebdb750936d"
server_key_iv: "071af9328c42e8b65bbeb11886236c27"
server_key_version: "encryptionkey@0"
I20260620 14:51:12.844708 8707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "5af9008904e54b6b88c7acedc852babd"
format_stamp: "Formatted at 2026-06-20 14:51:12 on dist-test-slave-2m5f"
server_key: "7afa8984ef1a1c1e6e2a7ebdb750936d"
server_key_iv: "071af9328c42e8b65bbeb11886236c27"
server_key_version: "encryptionkey@0"
I20260620 14:51:12.846148 8707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:12.847038 8722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:12.847230 8707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:12.847334 8707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "5af9008904e54b6b88c7acedc852babd"
format_stamp: "Formatted at 2026-06-20 14:51:12 on dist-test-slave-2m5f"
server_key: "7afa8984ef1a1c1e6e2a7ebdb750936d"
server_key_iv: "071af9328c42e8b65bbeb11886236c27"
server_key_version: "encryptionkey@0"
I20260620 14:51:12.847417 8707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:12.864567 8707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:12.866828 8707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:12.867009 8707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:12.867316 8707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:12.867650 8707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:12.867704 8707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:12.867741 8707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:12.867774 8707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:12.874735 8707 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:35263
I20260620 14:51:12.874776 8835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:35263 every 8 connection(s)
I20260620 14:51:12.875134 8707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:51:12 dist-test-slave-2m5f krb5kdc[8600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967072, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:12.882000 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 8707
I20260620 14:51:12.882102 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:12.882326 5594 external_mini_cluster.cc:1428] Setting key 50d0a3aec5303634440054979d7ab947
I20260620 14:51:12.885439 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:12.875933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:50489 (local address 127.5.118.190:42153)
0620 14:51:12.876075 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:12.876077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:12.876266 (+ 189us) server_negotiation.cc:407] Connection header received
0620 14:51:12.876777 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:12.876777 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:12.876790 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:12.876882 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:12.877894 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:12.878411 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:12.879427 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:12.879615 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:12.879674 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:12.881965 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:12.881966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:12.881966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:12.881974 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:12.883725 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:12.884306 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:12.884307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:12.884307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:12.884393 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:12.884682 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:12.884683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:12.884684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:12.884848 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:12.884918 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:12.885200 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:51:12.885304 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260620 14:51:12.885457 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:12.875929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:50489)
0620 14:51:12.876140 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:12.876191 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:12.876679 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:12.876940 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:12.876944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:12.877534 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:12.877785 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:12.877790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:12.878493 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:12.878494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:12.879345 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:12.879354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:12.879695 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:12.879696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:12.879836 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:12.880394 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:12.880403 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:12.881849 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:12.883815 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:12.883824 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:12.883827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:12.884214 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:12.884478 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:12.884478 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:12.884479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:12.884601 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:12.884912 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:12.884916 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:12.885042 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 14:51:12.885131 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:51:12.885279 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260620 14:51:12.886109 8836 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42153
I20260620 14:51:12.886229 8836 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:12.886422 8836 heartbeater.cc:511] Master 127.5.118.190:42153 requested a full tablet report, sending...
I20260620 14:51:12.886971 8645 ts_manager.cc:194] Registered new tserver with Master: 5af9008904e54b6b88c7acedc852babd (127.5.118.129:35263)
I20260620 14:51:12.887656 8645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:50489
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
I20260620 14:51:13.888773 8836 heartbeater.cc:503] Master 127.5.118.190:42153 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
W20260620 14:51:14.404362 5594 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.118.130: real 1.521s user 0.000s sys 0.003s
I20260620 14:51:14.404871 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:14.492610 8842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:14.492828 8842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:14.492854 8842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:14.494019 8842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:14.494063 8842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:14.494091 8842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:14.495559 8842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:14.495810 8842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:14.496065 8842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:14.501645 8850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:14.501637 8848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:14.501633 8847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:14.501904 8842 server_base.cc:1034] running on GCE node
I20260620 14:51:14.502261 8842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:14.502475 8842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:14.503652 8842 hybrid_clock.cc:648] HybridClock initialized: now 1781967074503625 us; error 37 us; skew 500 ppm
Jun 20 14:51:14 dist-test-slave-2m5f krb5kdc[8600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967074, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:14.506414 8842 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:51:14.506944 8842 webserver.cc:460] Webserver started at http://127.5.118.130:42811/ using document root <none> and password file <none>
I20260620 14:51:14.507185 8842 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:14.507259 8842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:14.507416 8842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:14.508603 8842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "ab81ff52775c477fb20cf2b884d2e394"
format_stamp: "Formatted at 2026-06-20 14:51:14 on dist-test-slave-2m5f"
server_key: "8dcca5f70f22ed6ee22fa9a352e73d1b"
server_key_iv: "2ed80257de35d272234ebf22be9dde0c"
server_key_version: "encryptionkey@0"
I20260620 14:51:14.508989 8842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "ab81ff52775c477fb20cf2b884d2e394"
format_stamp: "Formatted at 2026-06-20 14:51:14 on dist-test-slave-2m5f"
server_key: "8dcca5f70f22ed6ee22fa9a352e73d1b"
server_key_iv: "2ed80257de35d272234ebf22be9dde0c"
server_key_version: "encryptionkey@0"
I20260620 14:51:14.510432 8842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:14.511301 8857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:14.511492 8842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:14.511600 8842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "ab81ff52775c477fb20cf2b884d2e394"
format_stamp: "Formatted at 2026-06-20 14:51:14 on dist-test-slave-2m5f"
server_key: "8dcca5f70f22ed6ee22fa9a352e73d1b"
server_key_iv: "2ed80257de35d272234ebf22be9dde0c"
server_key_version: "encryptionkey@0"
I20260620 14:51:14.511675 8842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:14.529760 8842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:14.532311 8842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:14.532483 8842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:14.532752 8842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:14.533118 8842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:14.533172 8842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:14.533208 8842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:14.533236 8842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:14.539332 8842 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:35257
I20260620 14:51:14.539378 8970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:35257 every 8 connection(s)
I20260620 14:51:14.539760 8842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:14.540318 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 8842
I20260620 14:51:14.540454 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:14.540639 5594 external_mini_cluster.cc:1428] Setting key a7e68fdd2508c744c805838978cd1731
Jun 20 14:51:14 dist-test-slave-2m5f krb5kdc[8600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967074, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
I20260620 14:51:14.551672 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:14.540636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:48477 (local address 127.5.118.190:42153)
0620 14:51:14.540822 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:14.540825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:14.540882 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:51:14.541522 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:14.541523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:14.541534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:14.541601 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:14.542979 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:14.543894 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:14.545012 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:14.545293 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:14.545360 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:14.547990 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:14.547992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:14.547993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:14.548002 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:14.549826 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:14.550418 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:14.550420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:14.550420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:14.550493 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:14.550865 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:14.550867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:14.550867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:14.551018 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:14.551114 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:14.551495 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 14:51:14.551588 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":49,"threads_started":1}
I20260620 14:51:14.552022 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:14.540536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:48477)
0620 14:51:14.540739 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:14.540794 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:14.541386 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:14.541687 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:14.541692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:14.542563 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:14.542851 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:14.542858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:14.543997 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:14.543998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:14.544918 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:14.544921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:14.545394 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:14.545395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:14.545550 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:14.546157 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:14.546169 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:14.547875 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:14.549951 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:14.549965 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:14.549967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:14.550350 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:14.550653 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:14.550654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:14.550654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:14.550776 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:14.551134 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:14.551141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:14.551340 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 14:51:14.551466 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:51:14.551805 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260620 14:51:14.552627 8971 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42153
I20260620 14:51:14.552728 8971 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:14.552882 8971 heartbeater.cc:511] Master 127.5.118.190:42153 requested a full tablet report, sending...
I20260620 14:51:14.553287 8645 ts_manager.cc:194] Registered new tserver with Master: ab81ff52775c477fb20cf2b884d2e394 (127.5.118.130:35257)
I20260620 14:51:14.553686 8645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:48477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
W20260620 14:51:14.826527 5594 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.118.131: real 0.285s user 0.000s sys 0.003s
I20260620 14:51:14.826879 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:14.911458 8977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:14.911676 8977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:14.911701 8977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:14.912925 8977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:14.912969 8977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:14.912997 8977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:14.914565 8977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:14.914778 8977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:14.914959 8977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:14.920574 8982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:14.920562 8983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:14.920593 8977 server_base.cc:1034] running on GCE node
W20260620 14:51:14.920564 8985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:14.921260 8977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:14.921568 8977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:14.922761 8977 hybrid_clock.cc:648] HybridClock initialized: now 1781967074922744 us; error 29 us; skew 500 ppm
Jun 20 14:51:14 dist-test-slave-2m5f krb5kdc[8600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967074, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:14.925591 8977 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:51:14.926013 8977 webserver.cc:460] Webserver started at http://127.5.118.131:33173/ using document root <none> and password file <none>
I20260620 14:51:14.926215 8977 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:14.926259 8977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:14.926350 8977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:14.927414 8977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "f3a315222a9b42959f821f12e7180837"
format_stamp: "Formatted at 2026-06-20 14:51:14 on dist-test-slave-2m5f"
server_key: "d42ecf51487cf95f17b0b8cbb5816a8b"
server_key_iv: "2d2351c4395013e139923c4cf65cd442"
server_key_version: "encryptionkey@0"
I20260620 14:51:14.927711 8977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "f3a315222a9b42959f821f12e7180837"
format_stamp: "Formatted at 2026-06-20 14:51:14 on dist-test-slave-2m5f"
server_key: "d42ecf51487cf95f17b0b8cbb5816a8b"
server_key_iv: "2d2351c4395013e139923c4cf65cd442"
server_key_version: "encryptionkey@0"
I20260620 14:51:14.929131 8977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:14.929957 8992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:14.930153 8977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:14.930269 8977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "f3a315222a9b42959f821f12e7180837"
format_stamp: "Formatted at 2026-06-20 14:51:14 on dist-test-slave-2m5f"
server_key: "d42ecf51487cf95f17b0b8cbb5816a8b"
server_key_iv: "2d2351c4395013e139923c4cf65cd442"
server_key_version: "encryptionkey@0"
I20260620 14:51:14.930325 8977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:14.948732 8977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:14.951236 8977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:14.951429 8977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:14.951690 8977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:14.952020 8977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:14.952077 8977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:14.952121 8977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:14.952148 8977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:14.958441 8977 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:34357
I20260620 14:51:14.958479 9105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:34357 every 8 connection(s)
I20260620 14:51:14.958948 8977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:14.962168 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 8977
I20260620 14:51:14.962285 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:14.962482 5594 external_mini_cluster.cc:1428] Setting key fe04e57b6256d3753d9a92e19fab40a1
Jun 20 14:51:14 dist-test-slave-2m5f krb5kdc[8600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967074, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:14.970273 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:14.960030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:44721 (local address 127.5.118.190:42153)
0620 14:51:14.960164 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:14.960166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:14.960460 (+ 294us) server_negotiation.cc:407] Connection header received
0620 14:51:14.961014 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:14.961015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:14.961026 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:14.961091 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:14.962329 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:14.963267 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:14.964559 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:14.964779 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:14.964847 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:14.967142 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:14.967144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:14.967145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:14.967153 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:14.968958 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:14.969340 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:14.969341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:14.969341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:14.969396 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:14.969612 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:14.969612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:14.969613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:14.969790 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:14.969847 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:14.970123 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 14:51:14.970205 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:51:14.970387 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:14.959994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:44721)
0620 14:51:14.960304 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:14.960359 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:14.960892 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:14.961214 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:14.961218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:14.961825 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:14.962180 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:14.962189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:14.963420 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:14.963421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:14.964480 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:14.964483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:14.964883 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:14.964884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:14.964999 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:14.965541 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:14.965549 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:14.967051 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:14.969028 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:14.969035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:14.969036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:14.969274 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:14.969453 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:14.969454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:14.969454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:14.969556 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:14.969850 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:14.969854 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:14.969975 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 14:51:14.970063 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:51:14.970188 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":248,"threads_started":1}
I20260620 14:51:14.970862 9106 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42153
I20260620 14:51:14.970958 9106 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:14.971103 9106 heartbeater.cc:511] Master 127.5.118.190:42153 requested a full tablet report, sending...
I20260620 14:51:14.971482 8645 ts_manager.cc:194] Registered new tserver with Master: f3a315222a9b42959f821f12e7180837 (127.5.118.131:34357)
I20260620 14:51:14.971824 8645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:44721
I20260620 14:51:14.974502 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:51:14 dist-test-slave-2m5f krb5kdc[8600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967074, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:51:14 dist-test-slave-2m5f krb5kdc[8600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967074, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:14.998440 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:14.990950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.5.118.190:42153)
0620 14:51:14.991008 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:14.991010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:14.991134 (+ 124us) server_negotiation.cc:407] Connection header received
0620 14:51:14.991186 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:14.991187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:14.991197 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:14.991259 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:14.991821 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:14.992401 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:14.993200 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:14.993415 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:14.993484 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:14.995771 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:14.995773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:14.995773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:14.995781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:14.997250 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:14.997605 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:14.997607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:14.997607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:14.997669 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:14.997921 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:14.997922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:14.997923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:14.998106 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:14.998200 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:14.998305 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:51:14.998372 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 14:51:14.998819 8645 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49178
I20260620 14:51:14.999372 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 8707
I20260620 14:51:15.003759 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 8842
I20260620 14:51:15.008306 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 8977
I20260620 14:51:15.012315 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 8613
2026-06-20T14:51:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5025 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:15 dist-test-slave-2m5f krb5kdc[9118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:15 dist-test-slave-2m5f krb5kdc[9118](info): set up 2 sockets
Jun 20 14:51:15 dist-test-slave-2m5f krb5kdc[9118](info): commencing operation
krb5kdc: starting...
W20260620 14:51:16.056754 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:51:16.179831 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:40411
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:40411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:16.263171 9131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:16.263399 9131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:16.263423 9131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:16.264742 9131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:16.264791 9131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:16.264806 9131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:16.264818 9131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:16.264830 9131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:16.266440 9131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:40411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:40411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:16.266707 9131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:16.266973 9131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:16.271919 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:16.271920 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:16.271941 9139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:16.272341 9131 server_base.cc:1034] running on GCE node
I20260620 14:51:16.272545 9131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:16.272816 9131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:16.273964 9131 hybrid_clock.cc:648] HybridClock initialized: now 1781967076273958 us; error 42 us; skew 500 ppm
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:16.276589 9131 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:51:16.277050 9131 webserver.cc:460] Webserver started at http://127.5.118.190:46039/ using document root <none> and password file <none>
I20260620 14:51:16.277289 9131 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:16.277334 9131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:16.277480 9131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:16.278674 9131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "1861ebebba1d447e86a26c98bb3beb0f"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "8696bfb2ecf7348ee86790dc7cf15880"
server_key_iv: "d6e41548ab3dcaa7ba504bd74fa658ba"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.279089 9131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "1861ebebba1d447e86a26c98bb3beb0f"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "8696bfb2ecf7348ee86790dc7cf15880"
server_key_iv: "d6e41548ab3dcaa7ba504bd74fa658ba"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.280555 9131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:16.281383 9146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:16.281630 9131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:16.281713 9131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "1861ebebba1d447e86a26c98bb3beb0f"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "8696bfb2ecf7348ee86790dc7cf15880"
server_key_iv: "d6e41548ab3dcaa7ba504bd74fa658ba"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.281805 9131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:16.297749 9131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:16.299957 9131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:16.300110 9131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:16.304522 9131 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:40411
I20260620 14:51:16.304610 9198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:40411 every 8 connection(s)
I20260620 14:51:16.305179 9131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:16.305446 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 9131
I20260620 14:51:16.305562 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:16.305768 5594 external_mini_cluster.cc:1428] Setting key acbc9598c6dd1ea4c24dbaf656db72aa
I20260620 14:51:16.306005 9199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:16.309132 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f: Bootstrap starting.
I20260620 14:51:16.309818 9199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:16.310119 9199 log.cc:824] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:16.310834 9199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f: No bootstrap required, opened a new log
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:16.313570 9199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1861ebebba1d447e86a26c98bb3beb0f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40411 } }
I20260620 14:51:16.313755 9199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:16.313807 9199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1861ebebba1d447e86a26c98bb3beb0f, State: Initialized, Role: FOLLOWER
I20260620 14:51:16.313938 9199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1861ebebba1d447e86a26c98bb3beb0f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40411 } }
I20260620 14:51:16.314009 9199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:16.314033 9199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:16.314107 9199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:16.314675 9199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1861ebebba1d447e86a26c98bb3beb0f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40411 } }
I20260620 14:51:16.314764 9199 leader_election.cc:302] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1861ebebba1d447e86a26c98bb3beb0f; no voters:
I20260620 14:51:16.314947 9199 leader_election.cc:288] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:16.315012 9204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:16.315181 9204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [term 1 LEADER]: Becoming Leader. State: Replica: 1861ebebba1d447e86a26c98bb3beb0f, State: Running, Role: LEADER
I20260620 14:51:16.315241 9199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:16.315356 9204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1861ebebba1d447e86a26c98bb3beb0f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40411 } }
I20260620 14:51:16.315718 9205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1861ebebba1d447e86a26c98bb3beb0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1861ebebba1d447e86a26c98bb3beb0f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40411 } } }
I20260620 14:51:16.315801 9205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:16.315699 9206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1861ebebba1d447e86a26c98bb3beb0f. Latest consensus state: current_term: 1 leader_uuid: "1861ebebba1d447e86a26c98bb3beb0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1861ebebba1d447e86a26c98bb3beb0f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40411 } } }
I20260620 14:51:16.316507 9210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:16.316649 9206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:16.317387 9210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:16.317901 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:16.306381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38892 (local address 127.5.118.190:40411)
0620 14:51:16.306589 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:16.306597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:16.306624 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:51:16.307239 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:16.307246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:16.307300 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:16.307457 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:16.308127 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.309104 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.309883 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.310407 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.310544 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:16.313236 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:16.313241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:16.313241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:16.313258 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:16.315514 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.316039 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.316043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.316043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.316146 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.316664 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.316665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.316666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.317001 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:16.317126 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:16.317555 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 14:51:16.317697 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":91,"threads_started":1}
I20260620 14:51:16.319083 9210 catalog_manager.cc:1347] Generated new cluster ID: b50c416f3328421ab3398a45b96494d0
I20260620 14:51:16.319141 9210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:16.332000 9210 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:16.332847 9210 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:16.339533 9210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1861ebebba1d447e86a26c98bb3beb0f: Generated new TSK 0
I20260620 14:51:16.339777 9210 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:51:16.371349 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40411
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:16.455608 9225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:16.455915 9225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:16.455943 9225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:16.457229 9225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:16.457281 9225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:16.457310 9225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:16.458925 9225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:16.459215 9225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:16.459466 9225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:16.464915 9231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:16.465063 9225 server_base.cc:1034] running on GCE node
W20260620 14:51:16.465137 9233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:16.464931 9230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:16.465425 9225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:16.465699 9225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:16.466917 9225 hybrid_clock.cc:648] HybridClock initialized: now 1781967076466890 us; error 45 us; skew 500 ppm
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:16.469674 9225 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:51:16.470103 9225 webserver.cc:460] Webserver started at http://127.5.118.129:42609/ using document root <none> and password file <none>
I20260620 14:51:16.470348 9225 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:16.470394 9225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:16.470539 9225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:16.471642 9225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "376f1e570bec4d0faa83af1518893989"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "c2aab46ae1827e2f27109992200bba7d"
server_key_iv: "b331da9770acf419642100c8540bc3e7"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.472013 9225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "376f1e570bec4d0faa83af1518893989"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "c2aab46ae1827e2f27109992200bba7d"
server_key_iv: "b331da9770acf419642100c8540bc3e7"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.473420 9225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:16.474305 9240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:16.474514 9225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:16.474588 9225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "376f1e570bec4d0faa83af1518893989"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "c2aab46ae1827e2f27109992200bba7d"
server_key_iv: "b331da9770acf419642100c8540bc3e7"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.474723 9225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:16.485327 9225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:16.487654 9225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:16.487783 9225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:16.487982 9225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:16.488370 9225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:16.488410 9225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:16.488435 9225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:16.488480 9225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:16.494586 9225 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:35563
I20260620 14:51:16.494629 9353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:35563 every 8 connection(s)
I20260620 14:51:16.495007 9225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:16.496495 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 9225
I20260620 14:51:16.496635 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:16.496842 5594 external_mini_cluster.cc:1428] Setting key e8809e40cba854050d3ab3b80a219057
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
I20260620 14:51:16.506316 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:16.495819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:41401 (local address 127.5.118.190:40411)
0620 14:51:16.495878 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:16.495880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:16.496095 (+ 215us) server_negotiation.cc:407] Connection header received
0620 14:51:16.497351 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:16.497352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:16.497361 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:16.497419 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:16.498144 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.498704 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.499776 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.500129 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.500250 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:16.502896 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:16.502898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:16.502899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:16.502908 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:16.504679 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.505276 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.505277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.505278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.505345 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.505629 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.505630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.505631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.505798 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:16.505882 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:16.506161 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:51:16.506245 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:51:16.506517 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:16.495788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40411 (local address 127.5.118.129:41401)
0620 14:51:16.495975 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:16.496013 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:16.497245 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:16.497501 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:16.497505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:16.497841 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:16.498036 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:16.498041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.498841 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.498843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:16.499692 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:16.499695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.500265 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.500266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:16.500377 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:16.500998 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:16.501007 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:16.502784 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:16.504829 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:16.504836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:16.504839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:16.505183 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:16.505438 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:16.505438 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:16.505439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:16.505551 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:16.505869 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:16.505874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:16.506012 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 14:51:16.506127 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:51:16.506264 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260620 14:51:16.507230 9354 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40411
I20260620 14:51:16.507330 9354 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:16.507546 9354 heartbeater.cc:511] Master 127.5.118.190:40411 requested a full tablet report, sending...
I20260620 14:51:16.508301 9163 ts_manager.cc:194] Registered new tserver with Master: 376f1e570bec4d0faa83af1518893989 (127.5.118.129:35563)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
I20260620 14:51:16.509428 9163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:41401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:51:16.526556 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40411
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:16.609407 9359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:16.609620 9359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:16.609644 9359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:16.610896 9359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:16.610941 9359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:16.610968 9359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:16.612530 9359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:16.612793 9359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:16.613037 9359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:51:16.618636 9359 server_base.cc:1034] running on GCE node
W20260620 14:51:16.618606 9364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:16.618618 9367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:16.618693 9365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:16.619030 9359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:16.619257 9359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:16.620394 9359 hybrid_clock.cc:648] HybridClock initialized: now 1781967076620381 us; error 29 us; skew 500 ppm
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:16.622993 9359 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:51:16.623407 9359 webserver.cc:460] Webserver started at http://127.5.118.130:37869/ using document root <none> and password file <none>
I20260620 14:51:16.623611 9359 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:16.623652 9359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:16.623747 9359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:16.624825 9359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "db32bd27e64b4c48aec6d8b9a5752158"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "0260e6b877933ba34500f8b9debf0ab8"
server_key_iv: "adc18e06efa24ed0d27415f39570751d"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.625116 9359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "db32bd27e64b4c48aec6d8b9a5752158"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "0260e6b877933ba34500f8b9debf0ab8"
server_key_iv: "adc18e06efa24ed0d27415f39570751d"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.626370 9359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:16.627030 9374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:16.627219 9359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:16.627319 9359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "db32bd27e64b4c48aec6d8b9a5752158"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "0260e6b877933ba34500f8b9debf0ab8"
server_key_iv: "adc18e06efa24ed0d27415f39570751d"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.627398 9359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:16.636258 9359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:16.638037 9359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:16.638203 9359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:16.638424 9359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:16.638747 9359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:16.638803 9359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:16.638847 9359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:16.638875 9359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:16.645057 9359 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:34679
I20260620 14:51:16.645100 9487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:34679 every 8 connection(s)
I20260620 14:51:16.645468 9359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:16.651899 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 9359
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:16.652037 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:16.652257 5594 external_mini_cluster.cc:1428] Setting key 284acc925db911896f2ad293f4952092
I20260620 14:51:16.655791 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:16.646334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:41457 (local address 127.5.118.190:40411)
0620 14:51:16.646383 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:16.646386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:16.646579 (+ 193us) server_negotiation.cc:407] Connection header received
0620 14:51:16.647604 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:16.647606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:16.647619 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:16.647682 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:16.648420 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.648918 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.649935 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.650178 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.650305 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:16.652599 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:16.652601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:16.652601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:16.652610 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:16.654322 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.654769 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.654770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.654771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.654840 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.655099 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.655100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.655100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.655242 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:16.655319 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:16.655596 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:51:16.655688 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:51:16.656026 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:16.646292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40411 (local address 127.5.118.130:41457)
0620 14:51:16.646440 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:16.646488 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:16.647480 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:16.647768 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:16.647773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:16.648046 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:16.648317 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:16.648322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.649013 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.649014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:16.649839 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:16.649841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.650316 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.650317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:16.650459 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:16.650994 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:16.651002 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:16.652519 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:16.654412 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:16.654418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:16.654420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:16.654651 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:16.654911 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:16.654912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:16.654913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:16.655028 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:16.655321 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:16.655328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:16.655464 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 14:51:16.655577 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:51:16.655783 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260620 14:51:16.656607 9488 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40411
I20260620 14:51:16.656716 9488 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:16.656924 9488 heartbeater.cc:511] Master 127.5.118.190:40411 requested a full tablet report, sending...
I20260620 14:51:16.657352 9163 ts_manager.cc:194] Registered new tserver with Master: db32bd27e64b4c48aec6d8b9a5752158 (127.5.118.130:34679)
I20260620 14:51:16.657953 9163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:41457
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:51:16.677309 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40411
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:16.765861 9493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:16.766072 9493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:16.766098 9493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:16.767277 9493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:16.767320 9493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:16.767347 9493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:16.769047 9493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:16.769315 9493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:16.769562 9493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:16.775170 9499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:16.775209 9501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:16.775210 9493 server_base.cc:1034] running on GCE node
W20260620 14:51:16.775170 9498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:16.775646 9493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:16.775905 9493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:16.777068 9493 hybrid_clock.cc:648] HybridClock initialized: now 1781967076777087 us; error 64 us; skew 500 ppm
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:16.779608 9493 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:51:16.780217 9493 webserver.cc:460] Webserver started at http://127.5.118.131:38995/ using document root <none> and password file <none>
I20260620 14:51:16.780510 9493 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:16.780587 9493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:16.780741 9493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:16.781885 9493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "d5d6a717afff4687b190b69c3073240e"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "5accaa7b631d0bae1a063bf61090afc2"
server_key_iv: "b0ef1c4147cc01a37d798620ab9a081e"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.782222 9493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "d5d6a717afff4687b190b69c3073240e"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "5accaa7b631d0bae1a063bf61090afc2"
server_key_iv: "b0ef1c4147cc01a37d798620ab9a081e"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.783586 9493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:16.784409 9508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:16.784582 9493 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:16.784657 9493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "d5d6a717afff4687b190b69c3073240e"
format_stamp: "Formatted at 2026-06-20 14:51:16 on dist-test-slave-2m5f"
server_key: "5accaa7b631d0bae1a063bf61090afc2"
server_key_iv: "b0ef1c4147cc01a37d798620ab9a081e"
server_key_version: "encryptionkey@0"
I20260620 14:51:16.784723 9493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:16.797281 9493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:16.799650 9493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:16.799770 9493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:16.799962 9493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:16.800376 9493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:16.800416 9493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:16.800439 9493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:16.800484 9493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:16.806464 9493 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:41651
I20260620 14:51:16.806545 9621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:41651 every 8 connection(s)
I20260620 14:51:16.806890 9493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:16.812875 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 9493
I20260620 14:51:16.813009 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:16.813220 5594 external_mini_cluster.cc:1428] Setting key 70e6805149372184302c11dc3aba85e8
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:16.817807 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:16.807666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:46271 (local address 127.5.118.190:40411)
0620 14:51:16.807737 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:16.807739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:16.808196 (+ 457us) server_negotiation.cc:407] Connection header received
0620 14:51:16.809224 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:16.809225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:16.809237 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:16.809299 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:16.810010 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.810587 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.811732 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.811948 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.812014 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:16.814756 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:16.814758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:16.814758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:16.814766 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:16.816375 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.816811 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.816812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.816812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.816873 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.817141 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.817142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.817143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.817275 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:16.817331 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:16.817612 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 14:51:16.817712 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:51:16.818010 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:16.807647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40411 (local address 127.5.118.131:46271)
0620 14:51:16.807888 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:16.808059 (+ 171us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:16.809100 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:16.809382 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:16.809386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:16.809659 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:16.809859 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:16.809863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.810769 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.810770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:16.811624 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:16.811627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.812034 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.812035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:16.812147 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:16.812847 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:16.812856 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:16.814669 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:16.816457 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:16.816463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:16.816464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:16.816698 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:16.816939 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:16.816940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:16.816940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:16.817092 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:16.817336 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:16.817341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:16.817468 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 14:51:16.817587 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:51:16.817762 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
I20260620 14:51:16.818526 9622 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40411
I20260620 14:51:16.818615 9622 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:16.818768 9622 heartbeater.cc:511] Master 127.5.118.190:40411 requested a full tablet report, sending...
I20260620 14:51:16.819135 9163 ts_manager.cc:194] Registered new tserver with Master: d5d6a717afff4687b190b69c3073240e (127.5.118.131:41651)
I20260620 14:51:16.819478 9163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:46271
I20260620 14:51:16.825317 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.129@KRBTEST.COM
I20260620 14:51:16.851897 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:16.842910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.5.118.129:35563)
0620 14:51:16.843151 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:16.843156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:16.843168 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:51:16.843194 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:16.843196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:16.843211 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:16.843284 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:16.844043 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.844918 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.845711 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.846048 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.846117 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:16.848651 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:16.848665 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:16.848666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:16.848674 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:16.850342 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.850750 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.850754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.850755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.850827 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.851179 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.851180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.851181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.851395 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:16.851580 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:16.851667 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:51:16.851775 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:16.860522 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:16.853366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.5.118.190:40411)
0620 14:51:16.853411 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:16.853413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:16.853444 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:51:16.853497 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:16.853498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:16.853506 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:16.853550 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:16.854038 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.854512 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.855216 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:16.855410 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:16.855471 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:16.857930 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:16.857932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:16.857932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:16.857948 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:16.859272 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.859648 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.859650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.859650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.859711 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:16.859921 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:16.859922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:16.859922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:16.860076 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:16.860230 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:16.860330 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:51:16.860413 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:51:16.860968 9163 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38904
I20260620 14:51:16.861423 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 9225
I20260620 14:51:16.866325 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 9359
I20260620 14:51:16.870939 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 9493
I20260620 14:51:16.875298 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 9131
2026-06-20T14:51:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1863 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9634](info): set up 2 sockets
Jun 20 14:51:16 dist-test-slave-2m5f krb5kdc[9634](info): commencing operation
krb5kdc: starting...
W20260620 14:51:17.992650 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:18 dist-test-slave-2m5f krb5kdc[9634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:51:18.114980 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:46199
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:46199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:18.199600 9647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:18.199812 9647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:18.199836 9647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:18.200984 9647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:18.201022 9647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:18.201036 9647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:18.201049 9647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:18.201061 9647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:18.202543 9647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:46199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:46199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:18.202752 9647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:18.202932 9647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:18.208020 9655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:18.208003 9653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:18.208041 9647 server_base.cc:1034] running on GCE node
W20260620 14:51:18.208002 9652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:18.208516 9647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:18.208933 9647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:18.210150 9647 hybrid_clock.cc:648] HybridClock initialized: now 1781967078210132 us; error 29 us; skew 500 ppm
Jun 20 14:51:18 dist-test-slave-2m5f krb5kdc[9634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967078, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:18.219767 9647 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:51:18.220248 9647 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:18.220323 9647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:18.220486 9647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:18.221669 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "86230cf63c464c3b85cdb56f30bcfc30"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "0d88d38a48df9fcf1d1c7b3fd8ce4a74"
server_key_iv: "69cf1b704bfb7c995a801686bf7d224f"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.222049 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "86230cf63c464c3b85cdb56f30bcfc30"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "0d88d38a48df9fcf1d1c7b3fd8ce4a74"
server_key_iv: "69cf1b704bfb7c995a801686bf7d224f"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.223460 9647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:18.224642 9661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:18.224999 9647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:18.225113 9647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "86230cf63c464c3b85cdb56f30bcfc30"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "0d88d38a48df9fcf1d1c7b3fd8ce4a74"
server_key_iv: "69cf1b704bfb7c995a801686bf7d224f"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.225214 9647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:18.245682 9647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:18.247984 9647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:18.248226 9647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:18.252715 9647 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:46199
I20260620 14:51:18.252737 9713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:46199 every 8 connection(s)
I20260620 14:51:18.253067 9647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:18.254151 9714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:18.256781 9714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30: Bootstrap starting.
I20260620 14:51:18.257495 9714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:18.257844 9714 log.cc:824] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:18.258705 9714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30: No bootstrap required, opened a new log
I20260620 14:51:18.260322 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 9647
I20260620 14:51:18.260421 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:18.260609 5594 external_mini_cluster.cc:1428] Setting key 27a2f9a062f5b5e537365115f2e4605e
I20260620 14:51:18.260691 9714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86230cf63c464c3b85cdb56f30bcfc30" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46199 } }
I20260620 14:51:18.260893 9714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:18.260977 9714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86230cf63c464c3b85cdb56f30bcfc30, State: Initialized, Role: FOLLOWER
I20260620 14:51:18.261202 9714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86230cf63c464c3b85cdb56f30bcfc30" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46199 } }
I20260620 14:51:18.261312 9714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:18.261355 9714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:18.261451 9714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:18.262183 9714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86230cf63c464c3b85cdb56f30bcfc30" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46199 } }
I20260620 14:51:18.262313 9714 leader_election.cc:302] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86230cf63c464c3b85cdb56f30bcfc30; no voters:
I20260620 14:51:18.262511 9714 leader_election.cc:288] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:18.262567 9719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:18.262759 9719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [term 1 LEADER]: Becoming Leader. State: Replica: 86230cf63c464c3b85cdb56f30bcfc30, State: Running, Role: LEADER
I20260620 14:51:18.262866 9714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:18.263051 9719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86230cf63c464c3b85cdb56f30bcfc30" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46199 } }
I20260620 14:51:18.263509 9721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86230cf63c464c3b85cdb56f30bcfc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86230cf63c464c3b85cdb56f30bcfc30" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46199 } } }
I20260620 14:51:18.263532 9720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86230cf63c464c3b85cdb56f30bcfc30. Latest consensus state: current_term: 1 leader_uuid: "86230cf63c464c3b85cdb56f30bcfc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86230cf63c464c3b85cdb56f30bcfc30" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46199 } } }
I20260620 14:51:18.263682 9720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:18.263880 9721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:18.264024 9728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:18.264649 9728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:18.266952 9728 catalog_manager.cc:1347] Generated new cluster ID: adc4ef33962d475e9d4e2f6302afade8
I20260620 14:51:18.267020 9728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:51:18 dist-test-slave-2m5f krb5kdc[9634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:18.272301 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.261238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.5.118.190:46199)
0620 14:51:18.261450 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:18.261460 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:18.261494 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:51:18.261869 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:18.261875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:18.261924 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:18.262039 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:18.263000 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.264447 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.265509 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.266207 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.266334 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.268932 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:18.268936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:18.268937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:18.268949 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:18.270673 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:18.271065 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:18.271069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:18.271069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:18.271176 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:18.271387 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:18.271388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:18.271389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:18.271634 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:18.271828 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:18.271900 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 14:51:18.272043 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
I20260620 14:51:18.275898 9728 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:18.276754 9728 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:18.293268 9728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30: Generated new TSK 0
I20260620 14:51:18.293517 9728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:51:18.323554 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46199
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:18.406867 9740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:18.407116 9740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:18.407160 9740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:18.408480 9740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:18.408557 9740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:18.408603 9740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:18.410382 9740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:18.410687 9740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:18.410943 9740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:18.417151 9746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:18.417093 9748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:18.417310 9740 server_base.cc:1034] running on GCE node
W20260620 14:51:18.417104 9745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:18.417605 9740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:18.417893 9740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:18.419047 9740 hybrid_clock.cc:648] HybridClock initialized: now 1781967078419038 us; error 33 us; skew 500 ppm
Jun 20 14:51:18 dist-test-slave-2m5f krb5kdc[9634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967078, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:18.421804 9740 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:51:18.422194 9740 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:18.422245 9740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:18.422340 9740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:18.423372 9740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "bbdd0267a6354f6fa1c063a09dbd0481"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "62cc1841c64b8bbf4cf9bca23ba5522d"
server_key_iv: "c8295df9209722f8766a505031ea741b"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.423695 9740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "bbdd0267a6354f6fa1c063a09dbd0481"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "62cc1841c64b8bbf4cf9bca23ba5522d"
server_key_iv: "c8295df9209722f8766a505031ea741b"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.425143 9740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:51:18.425885 9754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:18.426090 9740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:18.426188 9740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "bbdd0267a6354f6fa1c063a09dbd0481"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "62cc1841c64b8bbf4cf9bca23ba5522d"
server_key_iv: "c8295df9209722f8766a505031ea741b"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.426263 9740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:18.453784 9740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:18.456328 9740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:18.456485 9740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:18.456817 9740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:18.457160 9740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:18.457219 9740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:18.457286 9740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:18.457316 9740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:18.463444 9740 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:39793
I20260620 14:51:18.463482 9867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:39793 every 8 connection(s)
I20260620 14:51:18.463780 9740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:18.469694 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 9740
I20260620 14:51:18.469820 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:18.470007 5594 external_mini_cluster.cc:1428] Setting key 48e6326bec61a19566d39688118f7807
Jun 20 14:51:18 dist-test-slave-2m5f krb5kdc[9634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967078, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:18.474224 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.464593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:47659 (local address 127.5.118.190:46199)
0620 14:51:18.464678 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:18.464681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:18.464908 (+ 227us) server_negotiation.cc:407] Connection header received
0620 14:51:18.465455 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:18.465456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:18.465473 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:18.465537 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:18.466515 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.467034 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.468063 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.468368 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.468445 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.471021 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:18.471022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:18.471023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:18.471030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:18.472780 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:18.473190 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:18.473191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:18.473192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:18.473250 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:18.473506 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:18.473507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:18.473508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:18.473635 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:18.473696 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:18.474033 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 14:51:18.474123 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:51:18.474426 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.464568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:46199 (local address 127.5.118.129:47659)
0620 14:51:18.464772 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:18.464817 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:18.465343 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:18.465612 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:18.465616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:18.465894 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:18.466397 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.466403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.467132 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.467133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.467965 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.467967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.468452 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.468453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.468591 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.469065 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:18.469072 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:18.470918 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:18.472878 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:18.472884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:18.472886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:18.473113 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:18.473323 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:18.473324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:18.473324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:18.473427 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:18.473753 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:18.473767 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:18.473895 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 14:51:18.474010 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:51:18.474176 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":103,"threads_started":1}
I20260620 14:51:18.474920 9868 heartbeater.cc:347] Connected to a master server at 127.5.118.190:46199
I20260620 14:51:18.475014 9868 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:18.475215 9868 heartbeater.cc:511] Master 127.5.118.190:46199 requested a full tablet report, sending...
I20260620 14:51:18.475852 9678 ts_manager.cc:194] Registered new tserver with Master: bbdd0267a6354f6fa1c063a09dbd0481 (127.5.118.129:39793)
I20260620 14:51:18.476877 9678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:47659
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:51:18.497345 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46199
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:18.579798 9873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:18.580045 9873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:18.580086 9873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:18.581307 9873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:18.581379 9873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:18.581422 9873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:18.582962 9873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:18.583257 9873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:18.583508 9873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:18.588682 9879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:18.588790 9873 server_base.cc:1034] running on GCE node
W20260620 14:51:18.588689 9878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:18.588775 9881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:18.589113 9873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:18.589393 9873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:18.590591 9873 hybrid_clock.cc:648] HybridClock initialized: now 1781967078590556 us; error 52 us; skew 500 ppm
Jun 20 14:51:18 dist-test-slave-2m5f krb5kdc[9634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967078, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:18.593282 9873 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:51:18.593640 9873 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:18.593713 9873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:18.593861 9873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:18.594959 9873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "b14ba311d350478581a0a53534cecbef"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "ecc137af2e5ca131c022b4e5c4e53ab1"
server_key_iv: "a527ea819bdc46a432c33545c99d8343"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.595336 9873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "b14ba311d350478581a0a53534cecbef"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "ecc137af2e5ca131c022b4e5c4e53ab1"
server_key_iv: "a527ea819bdc46a432c33545c99d8343"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.596825 9873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:51:18.597596 9887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:18.597777 9873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:18.597877 9873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "b14ba311d350478581a0a53534cecbef"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "ecc137af2e5ca131c022b4e5c4e53ab1"
server_key_iv: "a527ea819bdc46a432c33545c99d8343"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.597960 9873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:18.615494 9873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:18.617983 9873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:18.618158 9873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:18.618412 9873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:18.618732 9873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:18.618788 9873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:18.618846 9873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:18.618903 9873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:18.625042 9873 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:34023
I20260620 14:51:18.625079 10000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:34023 every 8 connection(s)
I20260620 14:51:18.625413 9873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:51:18 dist-test-slave-2m5f krb5kdc[9634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967078, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:18.632484 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 9873
I20260620 14:51:18.632606 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:18.632794 5594 external_mini_cluster.cc:1428] Setting key c6eb1d8504768b1bea089ecfeecf109b
I20260620 14:51:18.635521 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.626181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:41485 (local address 127.5.118.190:46199)
0620 14:51:18.626241 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:18.626243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:18.626462 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:51:18.626943 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:18.626944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:18.626954 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:18.627015 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:18.628041 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.628747 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.629847 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.630031 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.630088 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.632355 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:18.632356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:18.632356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:18.632363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:18.633976 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:18.634506 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:18.634507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:18.634507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:18.634555 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:18.634856 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:18.634857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:18.634857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:18.634970 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:18.635047 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:18.635349 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 14:51:18.635431 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:51:18.635732 10003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.626142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:46199 (local address 127.5.118.130:41485)
0620 14:51:18.626326 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:18.626371 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:18.626861 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:18.627071 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:18.627075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:18.627363 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:18.627949 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.627956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.628832 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.628833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.629769 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.629771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.630107 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.630108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.630230 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.630802 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:18.630810 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:18.632253 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:18.634059 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:18.634067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:18.634069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:18.634429 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:18.634619 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:18.634620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:18.634620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:18.634771 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:18.635052 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:18.635060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:18.635211 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 14:51:18.635317 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 14:51:18.635481 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260620 14:51:18.636320 10001 heartbeater.cc:347] Connected to a master server at 127.5.118.190:46199
I20260620 14:51:18.636404 10001 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:18.636571 10001 heartbeater.cc:511] Master 127.5.118.190:46199 requested a full tablet report, sending...
I20260620 14:51:18.636999 9678 ts_manager.cc:194] Registered new tserver with Master: b14ba311d350478581a0a53534cecbef (127.5.118.130:34023)
I20260620 14:51:18.637390 9678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:41485
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:51:18.659705 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46199
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:36497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:18.743866 10006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:18.744091 10006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:18.744117 10006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:18.745328 10006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:18.745376 10006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:18.745404 10006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:18.746886 10006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:18.747082 10006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:18.747277 10006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:18.752775 10012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:18.752837 10006 server_base.cc:1034] running on GCE node
W20260620 14:51:18.752784 10011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:18.752744 10014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:18.753288 10006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:18.753507 10006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:18.754649 10006 hybrid_clock.cc:648] HybridClock initialized: now 1781967078754636 us; error 29 us; skew 500 ppm
Jun 20 14:51:18 dist-test-slave-2m5f krb5kdc[9634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967078, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:18.757818 10006 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:51:18.758261 10006 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:18.758337 10006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:18.758472 10006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:18.759645 10006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "f9596ed144504c4c8de060c6811335ec"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "d9069c1957aeebc6c8dc4e8cdd86be13"
server_key_iv: "aebbd387df489f4f47720f865f9923d4"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.760124 10006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "f9596ed144504c4c8de060c6811335ec"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "d9069c1957aeebc6c8dc4e8cdd86be13"
server_key_iv: "aebbd387df489f4f47720f865f9923d4"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.762079 10006 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:51:18.763190 10020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:18.763367 10006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:51:18.763471 10006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "f9596ed144504c4c8de060c6811335ec"
format_stamp: "Formatted at 2026-06-20 14:51:18 on dist-test-slave-2m5f"
server_key: "d9069c1957aeebc6c8dc4e8cdd86be13"
server_key_iv: "aebbd387df489f4f47720f865f9923d4"
server_key_version: "encryptionkey@0"
I20260620 14:51:18.763552 10006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:18.771937 10006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:18.774103 10006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:18.774255 10006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:18.774507 10006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:18.774855 10006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:18.774892 10006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:18.774935 10006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:18.774981 10006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:18.781133 10006 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:46803
I20260620 14:51:18.781206 10133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:46803 every 8 connection(s)
I20260620 14:51:18.781491 10006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:18.784929 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 10006
I20260620 14:51:18.785055 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:18.785225 5594 external_mini_cluster.cc:1428] Setting key f32cb6337d84c1ece2f664a6f7ac9439
Jun 20 14:51:18 dist-test-slave-2m5f krb5kdc[9634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967078, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:18.791553 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.782241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:33107 (local address 127.5.118.190:46199)
0620 14:51:18.782286 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:18.782289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:18.782479 (+ 190us) server_negotiation.cc:407] Connection header received
0620 14:51:18.783053 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:18.783056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:18.783068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:18.783125 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:18.784125 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.784677 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.786659 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.786846 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.786905 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.789070 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:18.789071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:18.789071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:18.789079 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:18.790334 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:18.790720 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:18.790721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:18.790722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:18.790774 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:18.791009 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:18.791010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:18.791010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:18.791110 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:18.791162 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:18.791421 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:51:18.791485 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:51:18.791738 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.782207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:46199 (local address 127.5.118.131:33107)
0620 14:51:18.782342 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:18.782390 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:18.782940 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:18.783190 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:18.783194 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:18.783477 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:18.783974 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.783979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.785351 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.785352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.786530 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.786532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.786928 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.786929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.787038 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.787523 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:18.787531 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:18.788982 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:18.790407 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:18.790413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:18.790414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:18.790646 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:18.790834 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:18.790835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:18.790835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:18.790955 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:18.791180 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:18.791185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:18.791306 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 14:51:18.791392 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:51:18.791516 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
I20260620 14:51:18.792299 10134 heartbeater.cc:347] Connected to a master server at 127.5.118.190:46199
I20260620 14:51:18.792410 10134 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:18.792564 10134 heartbeater.cc:511] Master 127.5.118.190:46199 requested a full tablet report, sending...
I20260620 14:51:18.793012 9678 ts_manager.cc:194] Registered new tserver with Master: f9596ed144504c4c8de060c6811335ec (127.5.118.131:46803)
I20260620 14:51:18.793427 9678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:33107
I20260620 14:51:18.796947 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:51:18.804253 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.798240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.5.118.190:46199)
0620 14:51:18.798282 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:18.798285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:18.798397 (+ 112us) server_negotiation.cc:407] Connection header received
0620 14:51:18.798474 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:18.798475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:18.798485 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:18.798558 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:18.799125 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.799603 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.800428 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.800620 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.800726 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.801895 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:18.801896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:18.801897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:18.801904 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:18.803062 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:18.803410 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:18.803411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:18.803412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:18.803474 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:18.803680 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:18.803681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:18.803681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:18.803831 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:18.803940 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:18.804045 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:51:18.804129 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:51:18.805913 9678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:18.807406 9678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:18.814152 10146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.810088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:46803 (local address 127.0.0.1:46178)
0620 14:51:18.810494 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:18.810505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:18.810554 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:18.810770 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:18.810773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:18.810776 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:18.810863 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.810870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.812214 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.812215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.813092 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.813095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.813797 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.813798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.813890 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.813904 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:51:18.814004 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:51:18.814060 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":6,"thread_start_us":77,"threads_started":1}
I20260620 14:51:18.814152 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.810203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.5.118.131:46803)
0620 14:51:18.810359 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:18.810365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:18.810608 (+ 243us) server_negotiation.cc:407] Connection header received
0620 14:51:18.810650 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:18.810652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:18.810667 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:18.810740 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:18.810946 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.812075 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.813208 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.813707 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.813798 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.813860 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:18.814022 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:51:18.814060 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260620 14:51:18.814360 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.809881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:34023 (local address 127.0.0.1:45028)
0620 14:51:18.810228 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:18.810271 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:18.810339 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:18.810778 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:18.810779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:18.810781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:18.810870 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.810878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.811899 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.811900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.813117 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.813120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.814113 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.814115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.814190 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.814194 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:18.814244 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:51:18.814284 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":3,"thread_start_us":50,"threads_started":1}
I20260620 14:51:18.814555 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.810104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.5.118.130:34023)
0620 14:51:18.810505 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:18.810511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:18.810523 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:51:18.810566 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:18.810568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:18.810586 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:18.810676 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:18.810979 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.811800 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.813230 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.813967 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.814029 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.814080 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:18.814430 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 14:51:18.814478 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":71,"threads_started":1}
I20260620 14:51:18.815218 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.809975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39793 (local address 127.0.0.1:48212)
0620 14:51:18.810294 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:18.810305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:18.810359 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:18.811207 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:18.811208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:18.811211 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:18.811247 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.811248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.812767 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.812767 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.813809 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:18.813812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.814990 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.814991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:18.815052 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.815056 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:18.815114 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:51:18.815151 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":87,"threads_started":1}
I20260620 14:51:18.815382 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:18.810107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.5.118.129:39793)
0620 14:51:18.811035 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:18.811039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:18.811048 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:18.811084 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:18.811086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:18.811100 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:18.811177 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:18.811327 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.812398 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.813908 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:18.814719 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:18.814898 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:18.814963 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:18.815153 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 14:51:18.815204 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":97,"threads_started":1}
I20260620 14:51:18.815338 10068 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6f7774ef3b494292116764d3e5a68d (DEFAULT_TABLE table=test-table [id=026ff8af0d6d4c948f3131ebf692e388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:18.815362 9935 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6f7774ef3b494292116764d3e5a68d (DEFAULT_TABLE table=test-table [id=026ff8af0d6d4c948f3131ebf692e388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:18.815754 9935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6f7774ef3b494292116764d3e5a68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:18.815770 10068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6f7774ef3b494292116764d3e5a68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:18.816134 9802 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6f7774ef3b494292116764d3e5a68d (DEFAULT_TABLE table=test-table [id=026ff8af0d6d4c948f3131ebf692e388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:18.816480 9802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6f7774ef3b494292116764d3e5a68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:18.818346 10151 tablet_bootstrap.cc:492] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: Bootstrap starting.
I20260620 14:51:18.818905 10151 tablet_bootstrap.cc:654] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:18.819113 10152 tablet_bootstrap.cc:492] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: Bootstrap starting.
I20260620 14:51:18.819595 10153 tablet_bootstrap.cc:492] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481: Bootstrap starting.
I20260620 14:51:18.819700 10151 log.cc:824] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:18.819780 10152 tablet_bootstrap.cc:654] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:18.820261 10153 tablet_bootstrap.cc:654] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:18.820369 10152 log.cc:824] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:18.820662 10151 tablet_bootstrap.cc:492] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: No bootstrap required, opened a new log
I20260620 14:51:18.820748 10151 ts_tablet_manager.cc:1397] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:51:18.820905 10153 log.cc:824] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:18.821187 10152 tablet_bootstrap.cc:492] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: No bootstrap required, opened a new log
I20260620 14:51:18.821314 10152 ts_tablet_manager.cc:1397] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:51:18.821599 10153 tablet_bootstrap.cc:492] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481: No bootstrap required, opened a new log
I20260620 14:51:18.821683 10153 ts_tablet_manager.cc:1397] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:51:18.822811 10151 raft_consensus.cc:348] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 } } peers { permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 } } peers { permanent_uuid: "f9596ed144504c4c8de060c6811335ec" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46803 } }
I20260620 14:51:18.823000 10151 raft_consensus.cc:374] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:18.823038 10151 raft_consensus.cc:729] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b14ba311d350478581a0a53534cecbef, State: Initialized, Role: FOLLOWER
I20260620 14:51:18.823168 10151 consensus_queue.cc:260] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 } } peers { permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 } } peers { permanent_uuid: "f9596ed144504c4c8de060c6811335ec" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46803 } }
I20260620 14:51:18.823283 10152 raft_consensus.cc:348] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 } } peers { permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 } } peers { permanent_uuid: "f9596ed144504c4c8de060c6811335ec" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46803 } }
I20260620 14:51:18.823429 10152 raft_consensus.cc:374] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:18.823464 10151 ts_tablet_manager.cc:1428] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:18.823462 10152 raft_consensus.cc:729] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9596ed144504c4c8de060c6811335ec, State: Initialized, Role: FOLLOWER
I20260620 14:51:18.823348 10153 raft_consensus.cc:348] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 } } peers { permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 } } peers { permanent_uuid: "f9596ed144504c4c8de060c6811335ec" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46803 } }
I20260620 14:51:18.823508 10001 heartbeater.cc:503] Master 127.5.118.190:46199 was elected leader, sending a full tablet report...
I20260620 14:51:18.823529 10153 raft_consensus.cc:374] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:18.823563 10153 raft_consensus.cc:729] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdd0267a6354f6fa1c063a09dbd0481, State: Initialized, Role: FOLLOWER
I20260620 14:51:18.823608 10152 consensus_queue.cc:260] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 } } peers { permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 } } peers { permanent_uuid: "f9596ed144504c4c8de060c6811335ec" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46803 } }
I20260620 14:51:18.823662 10153 consensus_queue.cc:260] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 } } peers { permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 } } peers { permanent_uuid: "f9596ed144504c4c8de060c6811335ec" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46803 } }
I20260620 14:51:18.823909 10134 heartbeater.cc:503] Master 127.5.118.190:46199 was elected leader, sending a full tablet report...
I20260620 14:51:18.823956 10153 ts_tablet_manager.cc:1428] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:51:18.824115 9868 heartbeater.cc:503] Master 127.5.118.190:46199 was elected leader, sending a full tablet report...
I20260620 14:51:18.823909 10152 ts_tablet_manager.cc:1428] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20260620 14:51:18.876003 10002 tablet.cc:2369] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:51:18.964572 9869 tablet.cc:2369] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:51:19.032042 10135 tablet.cc:2369] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:19.133627 10158 raft_consensus.cc:482] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:19.133766 10158 raft_consensus.cc:504] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 } } peers { permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 } } peers { permanent_uuid: "f9596ed144504c4c8de060c6811335ec" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46803 } }
I20260620 14:51:19.134212 10158 leader_election.cc:288] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b14ba311d350478581a0a53534cecbef (127.5.118.130:34023), bbdd0267a6354f6fa1c063a09dbd0481 (127.5.118.129:39793)
I20260620 14:51:19.136930 10157 raft_consensus.cc:482] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:19.137076 10157 raft_consensus.cc:504] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 } } peers { permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 } } peers { permanent_uuid: "f9596ed144504c4c8de060c6811335ec" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46803 } }
I20260620 14:51:19.137889 10157 leader_election.cc:288] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbdd0267a6354f6fa1c063a09dbd0481 (127.5.118.129:39793), f9596ed144504c4c8de060c6811335ec (127.5.118.131:46803)
I20260620 14:51:19.137948 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.134404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39793 (local address 127.5.118.131:44501)
0620 14:51:19.134472 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.134487 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.134543 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.134751 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.134752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.134755 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:19.134835 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:19.134837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.135594 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.135595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:19.136760 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:19.136764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.137645 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.137646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:19.137744 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:19.137748 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.137798 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.137860 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:51:19.138213 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.134432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:34023 (local address 127.5.118.131:46827)
0620 14:51:19.134633 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.134643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.134703 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.134855 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.134856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.134857 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:19.134956 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:19.134958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.135849 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.135850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:19.136787 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:19.136789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.137693 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.137695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:19.137785 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:19.137789 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.138056 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.138110 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260620 14:51:19.138270 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.134494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:44501 (local address 127.5.118.129:39793)
0620 14:51:19.134543 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.134546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.134558 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:51:19.134596 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.134597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.134608 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.134681 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:19.134924 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.135478 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.136862 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.137440 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.137582 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:19.137593 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.138133 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.138188 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:51:19.138005 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.134541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:46827 (local address 127.5.118.130:34023)
0620 14:51:19.134590 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.134592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.134729 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:51:19.134760 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.134761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.134772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.134847 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:19.135069 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.135742 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.136952 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.137555 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.137646 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:19.137656 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.137886 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.137935 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:51:19.138679 9822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6f7774ef3b494292116764d3e5a68d" candidate_uuid: "f9596ed144504c4c8de060c6811335ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" is_pre_election: true
I20260620 14:51:19.138873 9822 raft_consensus.cc:2454] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9596ed144504c4c8de060c6811335ec in term 0.
I20260620 14:51:19.139009 9955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6f7774ef3b494292116764d3e5a68d" candidate_uuid: "f9596ed144504c4c8de060c6811335ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14ba311d350478581a0a53534cecbef" is_pre_election: true
I20260620 14:51:19.139119 9955 raft_consensus.cc:2454] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9596ed144504c4c8de060c6811335ec in term 0.
I20260620 14:51:19.139158 10022 leader_election.cc:302] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbdd0267a6354f6fa1c063a09dbd0481, f9596ed144504c4c8de060c6811335ec; no voters:
I20260620 14:51:19.139323 10158 raft_consensus.cc:2781] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:19.139393 10158 raft_consensus.cc:482] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:19.139415 10158 raft_consensus.cc:3037] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:19.140487 10158 raft_consensus.cc:504] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 } } peers { permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 } } peers { permanent_uuid: "f9596ed144504c4c8de060c6811335ec" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46803 } }
I20260620 14:51:19.140638 10158 leader_election.cc:288] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [CANDIDATE]: Term 1 election: Requested vote from peers b14ba311d350478581a0a53534cecbef (127.5.118.130:34023), bbdd0267a6354f6fa1c063a09dbd0481 (127.5.118.129:39793)
I20260620 14:51:19.140830 9955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6f7774ef3b494292116764d3e5a68d" candidate_uuid: "f9596ed144504c4c8de060c6811335ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b14ba311d350478581a0a53534cecbef"
I20260620 14:51:19.140913 9955 raft_consensus.cc:3037] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:19.140888 9822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6f7774ef3b494292116764d3e5a68d" candidate_uuid: "f9596ed144504c4c8de060c6811335ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbdd0267a6354f6fa1c063a09dbd0481"
I20260620 14:51:19.141023 9822 raft_consensus.cc:3037] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:19.141682 9955 raft_consensus.cc:2454] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9596ed144504c4c8de060c6811335ec in term 1.
I20260620 14:51:19.141896 10021 leader_election.cc:302] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b14ba311d350478581a0a53534cecbef, f9596ed144504c4c8de060c6811335ec; no voters:
I20260620 14:51:19.142009 10158 raft_consensus.cc:2781] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:19.142057 9822 raft_consensus.cc:2454] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9596ed144504c4c8de060c6811335ec in term 1.
I20260620 14:51:19.142143 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.137789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:46803 (local address 127.5.118.130:37383)
0620 14:51:19.138533 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.138545 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.138635 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.138824 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.138824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.138826 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:19.138900 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:19.138903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.139824 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.139825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:19.141066 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:19.141068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.141848 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.141848 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:19.141925 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:19.141930 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.141992 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.142040 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":63,"threads_started":1}
I20260620 14:51:19.142187 10158 raft_consensus.cc:686] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 1 LEADER]: Becoming Leader. State: Replica: f9596ed144504c4c8de060c6811335ec, State: Running, Role: LEADER
I20260620 14:51:19.142248 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.138032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:37383 (local address 127.5.118.131:46803)
0620 14:51:19.138078 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.138080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.138603 (+ 523us) server_negotiation.cc:407] Connection header received
0620 14:51:19.138718 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.138719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.138729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.138760 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:19.138988 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.139739 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.141163 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.141766 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.141849 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:19.141858 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.142087 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.142134 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:51:19.142318 10158 consensus_queue.cc:237] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 } } peers { permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 } } peers { permanent_uuid: "f9596ed144504c4c8de060c6811335ec" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46803 } }
I20260620 14:51:19.142714 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6f7774ef3b494292116764d3e5a68d" candidate_uuid: "b14ba311d350478581a0a53534cecbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9596ed144504c4c8de060c6811335ec" is_pre_election: true
I20260620 14:51:19.143114 9677 catalog_manager.cc:5495] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec reported cstate change: term changed from 0 to 1, leader changed from <none> to f9596ed144504c4c8de060c6811335ec (127.5.118.131). New cstate: current_term: 1 leader_uuid: "f9596ed144504c4c8de060c6811335ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9596ed144504c4c8de060c6811335ec" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46803 } health_report { overall_health: HEALTHY } } }
I20260620 14:51:19.143386 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.138012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:51639 (local address 127.5.118.129:39793)
0620 14:51:19.138353 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.138354 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.138362 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:19.138382 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.138383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.138393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.138595 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:19.138816 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.139532 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.142404 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.143011 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.143062 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:19.143071 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.143282 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.143327 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":64,"threads_started":1}
I20260620 14:51:19.143388 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.137777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39793 (local address 127.5.118.130:51639)
0620 14:51:19.138130 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.138141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.138189 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.138616 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.138617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.138620 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:19.138722 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:19.138724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.139621 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.139622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:19.140733 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:19.140737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.143139 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.143140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:19.143200 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:19.143203 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.143245 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.143278 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":24,"thread_start_us":89,"threads_started":1}
I20260620 14:51:19.143893 9822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6f7774ef3b494292116764d3e5a68d" candidate_uuid: "b14ba311d350478581a0a53534cecbef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" is_pre_election: true
I20260620 14:51:19.144027 9822 raft_consensus.cc:2379] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b14ba311d350478581a0a53534cecbef in current term 1: Already voted for candidate f9596ed144504c4c8de060c6811335ec in this term.
I20260620 14:51:19.144325 9889 leader_election.cc:302] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b14ba311d350478581a0a53534cecbef; no voters: bbdd0267a6354f6fa1c063a09dbd0481, f9596ed144504c4c8de060c6811335ec
I20260620 14:51:19.144482 10157 raft_consensus.cc:2726] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:51:19.172906 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.170583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.5.118.131:46803)
0620 14:51:19.170655 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.170657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.170666 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:19.170698 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.170699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.170716 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.170783 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:19.170951 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.171464 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.172220 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:19.172403 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:19.172462 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:19.172605 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:19.172714 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:19.172778 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.172810 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.172827 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:51:19.174777 9822 raft_consensus.cc:1261] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481 [term 1 FOLLOWER]: Refusing update from remote peer f9596ed144504c4c8de060c6811335ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:19.174796 9955 raft_consensus.cc:1261] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 1 FOLLOWER]: Refusing update from remote peer f9596ed144504c4c8de060c6811335ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:19.175166 10164 consensus_queue.cc:1035] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b14ba311d350478581a0a53534cecbef" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:19.175307 10158 consensus_queue.cc:1035] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbdd0267a6354f6fa1c063a09dbd0481" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:19.178419 10169 mvcc.cc:205] Tried to move back new op lower bound from 7298937156297064448 to 7298937156167421952. Current Snapshot: MvccSnapshot[applied={T|T < 7298937156297064448 or (T in {7298937156297064448})}]
I20260620 14:51:19.180006 9677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:19.180124 9677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:19.180485 10168 mvcc.cc:205] Tried to move back new op lower bound from 7298937156297064448 to 7298937156167421952. Current Snapshot: MvccSnapshot[applied={T|T < 7298937156297064448}]
I20260620 14:51:19.180917 9677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86230cf63c464c3b85cdb56f30bcfc30: Sending DeleteTablet for 3 replicas of tablet 4b6f7774ef3b494292116764d3e5a68d
I20260620 14:51:19.181236 9935 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b6f7774ef3b494292116764d3e5a68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:19 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:45028
I20260620 14:51:19.181273 10068 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b6f7774ef3b494292116764d3e5a68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:19 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:46178
I20260620 14:51:19.181458 10177 tablet_replica.cc:330] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: stopping tablet replica
I20260620 14:51:19.181614 10177 raft_consensus.cc:2229] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:19.181603 9802 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b6f7774ef3b494292116764d3e5a68d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:19 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:48212
I20260620 14:51:19.181802 10179 tablet_replica.cc:330] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481: stopping tablet replica
I20260620 14:51:19.181807 10177 raft_consensus.cc:2258] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:19.181943 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 9740
I20260620 14:51:19.182015 10179 raft_consensus.cc:2229] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:19.182121 10179 raft_consensus.cc:2258] T 4b6f7774ef3b494292116764d3e5a68d P bbdd0267a6354f6fa1c063a09dbd0481 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:19.182289 10177 ts_tablet_manager.cc:1905] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:19.183979 10177 ts_tablet_manager.cc:1918] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:19.184047 10177 log.cc:1196] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/4b6f7774ef3b494292116764d3e5a68d
I20260620 14:51:19.184373 10177 ts_tablet_manager.cc:1939] T 4b6f7774ef3b494292116764d3e5a68d P b14ba311d350478581a0a53534cecbef: Deleting consensus metadata
I20260620 14:51:19.184772 9662 catalog_manager.cc:4841] TS b14ba311d350478581a0a53534cecbef (127.5.118.130:34023): tablet 4b6f7774ef3b494292116764d3e5a68d (table test-table [id=026ff8af0d6d4c948f3131ebf692e388]) successfully deleted
I20260620 14:51:19.188383 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 9873
W20260620 14:51:19.188393 9663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:39793 (error 108)
I20260620 14:51:19.188716 10178 tablet_replica.cc:330] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: stopping tablet replica
I20260620 14:51:19.188858 10178 raft_consensus.cc:2229] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:19.188982 10178 raft_consensus.cc:2258] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:19.189394 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.189208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39793 (local address 127.0.0.1:48224)
0620 14:51:19.189307 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.189354 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 14:51:19.189543 9663 catalog_manager.cc:4584] TS bbdd0267a6354f6fa1c063a09dbd0481 (127.5.118.129:39793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b6f7774ef3b494292116764d3e5a68d: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39793: connect: Connection refused (error 111)
I20260620 14:51:19.189918 10178 ts_tablet_manager.cc:1905] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:19.191619 10178 ts_tablet_manager.cc:1918] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:19.191706 10178 log.cc:1196] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/4b6f7774ef3b494292116764d3e5a68d
I20260620 14:51:19.192014 10178 ts_tablet_manager.cc:1939] T 4b6f7774ef3b494292116764d3e5a68d P f9596ed144504c4c8de060c6811335ec: Deleting consensus metadata
I20260620 14:51:19.192543 9662 catalog_manager.cc:4841] TS f9596ed144504c4c8de060c6811335ec (127.5.118.131:46803): tablet 4b6f7774ef3b494292116764d3e5a68d (table test-table [id=026ff8af0d6d4c948f3131ebf692e388]) successfully deleted
I20260620 14:51:19.194808 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 10006
I20260620 14:51:19.199005 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 9647
2026-06-20T14:51:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2322 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:51:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:19Z Disabled control of system clock
I20260620 14:51:19.226425 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:40361
--webserver_interface=127.5.118.190
--webserver_port=0
--builtin_ntp_servers=127.5.118.148:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:51:19.308235 10184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:19.308454 10184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:19.308480 10184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:19.309605 10184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:19.309644 10184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:19.309664 10184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:19.309677 10184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:19.309689 10184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:19.311213 10184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:40361
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:19.311424 10184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:19.311777 10184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:19.316936 10190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:19.316993 10189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:19.316943 10192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:19.317023 10184 server_base.cc:1034] running on GCE node
I20260620 14:51:19.317358 10184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:19.317632 10184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:19.318785 10184 hybrid_clock.cc:648] HybridClock initialized: now 1781967079318761 us; error 37 us; skew 500 ppm
I20260620 14:51:19.320324 10184 webserver.cc:460] Webserver started at http://127.5.118.190:45373/ using document root <none> and password file <none>
I20260620 14:51:19.320596 10184 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:19.320645 10184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:19.320784 10184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:19.321884 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "8c01b2a057dc43b988062a86b016486f"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "67de9e6f2a26b140d9ec62cd64e31bcf"
server_key_iv: "cbf86df7e1a27cb26497d3bc38c24b6f"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.322263 10184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "8c01b2a057dc43b988062a86b016486f"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "67de9e6f2a26b140d9ec62cd64e31bcf"
server_key_iv: "cbf86df7e1a27cb26497d3bc38c24b6f"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.323613 10184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:19.324409 10198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:19.324651 10184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:19.324802 10184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "8c01b2a057dc43b988062a86b016486f"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "67de9e6f2a26b140d9ec62cd64e31bcf"
server_key_iv: "cbf86df7e1a27cb26497d3bc38c24b6f"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.324887 10184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:19.328493 10184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:19.328795 10184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:19.328948 10184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:19.332937 10184 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:40361
I20260620 14:51:19.332967 10250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:40361 every 8 connection(s)
I20260620 14:51:19.333348 10184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:19.334062 10251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:19.337069 10251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f: Bootstrap starting.
I20260620 14:51:19.337690 10251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:19.337991 10251 log.cc:824] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:19.338723 10251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f: No bootstrap required, opened a new log
I20260620 14:51:19.340515 10251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c01b2a057dc43b988062a86b016486f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40361 } }
I20260620 14:51:19.340674 10251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:19.340698 10251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c01b2a057dc43b988062a86b016486f, State: Initialized, Role: FOLLOWER
I20260620 14:51:19.340870 10251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c01b2a057dc43b988062a86b016486f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40361 } }
I20260620 14:51:19.340970 10251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:19.341009 10251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:19.341085 10251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:19.341104 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 10184
I20260620 14:51:19.341205 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:19.341391 5594 external_mini_cluster.cc:1428] Setting key 4df4b445000c9b6af3c648e74ec931e5
I20260620 14:51:19.341969 10251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c01b2a057dc43b988062a86b016486f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40361 } }
I20260620 14:51:19.342109 10251 leader_election.cc:302] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c01b2a057dc43b988062a86b016486f; no voters:
I20260620 14:51:19.342283 10251 leader_election.cc:288] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:19.342316 10256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:19.342418 10256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [term 1 LEADER]: Becoming Leader. State: Replica: 8c01b2a057dc43b988062a86b016486f, State: Running, Role: LEADER
I20260620 14:51:19.342559 10251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:19.342559 10256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c01b2a057dc43b988062a86b016486f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40361 } }
I20260620 14:51:19.343303 10257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c01b2a057dc43b988062a86b016486f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c01b2a057dc43b988062a86b016486f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40361 } } }
I20260620 14:51:19.343554 10257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:19.343492 10258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c01b2a057dc43b988062a86b016486f. Latest consensus state: current_term: 1 leader_uuid: "8c01b2a057dc43b988062a86b016486f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c01b2a057dc43b988062a86b016486f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40361 } } }
I20260620 14:51:19.344004 10258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:19.344318 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.341995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.5.118.190:40361)
0620 14:51:19.342256 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.342266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.342300 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:51:19.342444 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.342453 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.342504 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.343031 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:19.343613 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:19.343617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:19.343617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:19.343621 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:19.343663 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:19.343678 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:19.343969 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.344012 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.344073 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
W20260620 14:51:19.344643 10272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:51:19.344697 10272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:51:19.344753 10267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:19.344907 10267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:19.346401 10267 catalog_manager.cc:1347] Generated new cluster ID: e52dcb4cc6364489b5d5eb56babf06af
I20260620 14:51:19.346460 10267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:19.361163 10267 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:19.361397 10267 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:19.368347 10267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f: Generated new TSK 0
I20260620 14:51:19.368602 10267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:51:19.371481 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40361
--builtin_ntp_servers=127.5.118.148:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:51:19.454712 10275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:19.454941 10275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:19.454967 10275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:19.456293 10275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:19.456341 10275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:19.456370 10275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:19.457814 10275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:19.458083 10275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:19.458410 10275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:19.464296 10283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:19.464287 10281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:19.464287 10280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:19.464665 10275 server_base.cc:1034] running on GCE node
I20260620 14:51:19.464870 10275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:19.465138 10275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:19.466296 10275 hybrid_clock.cc:648] HybridClock initialized: now 1781967079466285 us; error 27 us; skew 500 ppm
I20260620 14:51:19.471040 10275 webserver.cc:460] Webserver started at http://127.5.118.129:35033/ using document root <none> and password file <none>
I20260620 14:51:19.471308 10275 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:19.471357 10275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:19.471511 10275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:19.472741 10275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "0b920a3821ab4d9e85599449f064b7e9"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "39ba2d8d19402c621d2b998236f67f97"
server_key_iv: "3849ca8935ae35d4e16b61d27b6cd16c"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.473104 10275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "0b920a3821ab4d9e85599449f064b7e9"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "39ba2d8d19402c621d2b998236f67f97"
server_key_iv: "3849ca8935ae35d4e16b61d27b6cd16c"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.474445 10275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:19.475245 10289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:19.475414 10275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:19.475524 10275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "0b920a3821ab4d9e85599449f064b7e9"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "39ba2d8d19402c621d2b998236f67f97"
server_key_iv: "3849ca8935ae35d4e16b61d27b6cd16c"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.475608 10275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:19.479024 10275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:19.479314 10275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:19.479449 10275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:19.479678 10275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:19.479988 10275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:19.480019 10275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:19.480075 10275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:19.480100 10275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:19.486056 10275 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:39181
I20260620 14:51:19.486109 10402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:39181 every 8 connection(s)
I20260620 14:51:19.486433 10275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:19.486689 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 10275
I20260620 14:51:19.486785 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:19.487010 5594 external_mini_cluster.cc:1428] Setting key 139007a7336a06483701b3a81cdc55bd
I20260620 14:51:19.488147 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40361
--builtin_ntp_servers=127.5.118.148:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:51:19.490023 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.487210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:46523 (local address 127.5.118.190:40361)
0620 14:51:19.487377 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.487380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.487670 (+ 290us) server_negotiation.cc:407] Connection header received
0620 14:51:19.488186 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.488187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.488194 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.488260 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:19.489750 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:19.489751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:19.489752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:19.489757 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:19.489773 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:19.489778 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:19.489838 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.489935 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.489953 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260620 14:51:19.490291 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.487222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40361 (local address 127.5.118.129:46523)
0620 14:51:19.487525 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.487569 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.488051 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.488402 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.488406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.488416 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:19.489543 (+ 1127us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:19.489544 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:19.489589 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:19.489632 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:19.489838 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:19.489844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:19.489847 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.489946 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.490010 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":92,"threads_started":1}
I20260620 14:51:19.490864 10403 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40361
I20260620 14:51:19.490973 10403 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:19.491124 10403 heartbeater.cc:511] Master 127.5.118.190:40361 requested a full tablet report, sending...
I20260620 14:51:19.491703 10215 ts_manager.cc:194] Registered new tserver with Master: 0b920a3821ab4d9e85599449f064b7e9 (127.5.118.129:39181)
W20260620 14:51:19.585937 10406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:19.586256 10406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:19.586292 10406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:19.588150 10406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:19.588241 10406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:19.588284 10406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:19.590649 10406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:19.591084 10406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:19.591485 10406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:19.597182 10414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:19.597150 10411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:19.597245 10412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:19.597396 10406 server_base.cc:1034] running on GCE node
I20260620 14:51:19.597693 10406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:19.597963 10406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:19.599134 10406 hybrid_clock.cc:648] HybridClock initialized: now 1781967079599126 us; error 32 us; skew 500 ppm
I20260620 14:51:19.600569 10406 webserver.cc:460] Webserver started at http://127.5.118.130:39311/ using document root <none> and password file <none>
I20260620 14:51:19.600850 10406 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:19.600925 10406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:19.601071 10406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:19.602140 10406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "5bc1fd856a984c0e8c2b629bc8cd6135"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "4e22172ec94f8d8833db810c73d27185"
server_key_iv: "9455bdfe9b0edf8aea7f95a95e52ea58"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.602448 10406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "5bc1fd856a984c0e8c2b629bc8cd6135"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "4e22172ec94f8d8833db810c73d27185"
server_key_iv: "9455bdfe9b0edf8aea7f95a95e52ea58"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.603807 10406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:19.604615 10420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:19.604784 10406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:51:19.604892 10406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "5bc1fd856a984c0e8c2b629bc8cd6135"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "4e22172ec94f8d8833db810c73d27185"
server_key_iv: "9455bdfe9b0edf8aea7f95a95e52ea58"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.604969 10406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:19.608656 10406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:19.608929 10406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:19.609071 10406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:19.609284 10406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:19.609606 10406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:19.609669 10406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:19.609712 10406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:19.609740 10406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:19.615839 10406 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:34375
I20260620 14:51:19.615880 10533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:34375 every 8 connection(s)
I20260620 14:51:19.616331 10406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:19.619344 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.617106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:38749 (local address 127.5.118.190:40361)
0620 14:51:19.617202 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.617205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.617356 (+ 151us) server_negotiation.cc:407] Connection header received
0620 14:51:19.617806 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.617807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.617812 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.617866 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:19.619081 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:19.619083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:19.619083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:19.619087 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:19.619103 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:19.619108 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:19.619181 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.619249 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.619261 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:51:19.619525 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.617056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40361 (local address 127.5.118.130:38749)
0620 14:51:19.617230 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.617263 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.617723 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.617922 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.617925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.617933 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:19.618913 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:19.618913 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:19.618946 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:19.618977 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:19.619182 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:19.619184 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:19.619186 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.619239 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.619289 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
I20260620 14:51:19.620074 10534 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40361
I20260620 14:51:19.620158 10534 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:19.620345 10534 heartbeater.cc:511] Master 127.5.118.190:40361 requested a full tablet report, sending...
I20260620 14:51:19.620725 10215 ts_manager.cc:194] Registered new tserver with Master: 5bc1fd856a984c0e8c2b629bc8cd6135 (127.5.118.130:34375)
I20260620 14:51:19.623740 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 10406
I20260620 14:51:19.623823 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:19.624006 5594 external_mini_cluster.cc:1428] Setting key 64083d04e365a7a219f1ab2659f85baf
I20260620 14:51:19.624987 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40361
--builtin_ntp_servers=127.5.118.148:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:51:19.708446 10537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:19.708671 10537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:19.708698 10537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:19.710013 10537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:19.710058 10537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:19.710086 10537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:19.711491 10537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40361
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:19.711768 10537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:19.712107 10537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:19.717445 10545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:19.717453 10542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:19.717475 10543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:19.717763 10537 server_base.cc:1034] running on GCE node
I20260620 14:51:19.718024 10537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:19.718307 10537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:19.719560 10537 hybrid_clock.cc:648] HybridClock initialized: now 1781967079719468 us; error 37 us; skew 500 ppm
I20260620 14:51:19.721102 10537 webserver.cc:460] Webserver started at http://127.5.118.131:32969/ using document root <none> and password file <none>
I20260620 14:51:19.721328 10537 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:19.721381 10537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:19.721482 10537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:19.722723 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "6c7b40e2f6bf45589dd5d21351ea8d46"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "cdf7300e399c0c7b909ba3b73d3f672c"
server_key_iv: "6a7db2430350733fcbe8abb2be6ddb27"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.723078 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "6c7b40e2f6bf45589dd5d21351ea8d46"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "cdf7300e399c0c7b909ba3b73d3f672c"
server_key_iv: "6a7db2430350733fcbe8abb2be6ddb27"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.724650 10537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:19.725492 10551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:19.725726 10537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:19.725843 10537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "6c7b40e2f6bf45589dd5d21351ea8d46"
format_stamp: "Formatted at 2026-06-20 14:51:19 on dist-test-slave-2m5f"
server_key: "cdf7300e399c0c7b909ba3b73d3f672c"
server_key_iv: "6a7db2430350733fcbe8abb2be6ddb27"
server_key_version: "encryptionkey@0"
I20260620 14:51:19.725924 10537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:19.729734 10537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:19.730037 10537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:19.730194 10537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:19.730432 10537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:19.730780 10537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:19.730835 10537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:19.730876 10537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:19.730904 10537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:19.737032 10537 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:46315
I20260620 14:51:19.737135 10664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:46315 every 8 connection(s)
I20260620 14:51:19.737452 10537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:19.740010 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 10537
I20260620 14:51:19.740096 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:19.740304 5594 external_mini_cluster.cc:1428] Setting key e7dd1a2413b62651bab1899d17154d06
I20260620 14:51:19.740796 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.738310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:39383 (local address 127.5.118.190:40361)
0620 14:51:19.738402 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.738404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.738588 (+ 184us) server_negotiation.cc:407] Connection header received
0620 14:51:19.739105 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.739106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.739112 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.739171 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:19.740543 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:19.740545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:19.740545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:19.740550 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:19.740565 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:19.740570 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:19.740624 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.740708 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.740726 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:51:19.741010 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.738203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40361 (local address 127.5.118.131:39383)
0620 14:51:19.738463 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.738499 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.738998 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.739254 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.739258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.739272 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:19.740384 (+ 1112us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:19.740385 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:19.740423 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:19.740453 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:19.740620 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:19.740625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:19.740629 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.740717 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.740780 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":106,"threads_started":1}
I20260620 14:51:19.741530 10665 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40361
I20260620 14:51:19.741637 10665 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:19.741806 10665 heartbeater.cc:511] Master 127.5.118.190:40361 requested a full tablet report, sending...
I20260620 14:51:19.742137 10215 ts_manager.cc:194] Registered new tserver with Master: 6c7b40e2f6bf45589dd5d21351ea8d46 (127.5.118.131:46315)
I20260620 14:51:19.752220 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:51:19.754752 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.753621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.5.118.190:40361)
0620 14:51:19.753700 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.753702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.753775 (+ 73us) server_negotiation.cc:407] Connection header received
0620 14:51:19.753816 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.753817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.753822 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.753865 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:19.754534 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:19.754535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:19.754535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:19.754538 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:19.754551 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:19.754554 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:19.754595 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.754634 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.754646 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:51:19.756379 10215 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:19.757933 10215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:19.762300 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.760535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39181 (local address 127.0.0.1:35282)
0620 14:51:19.760947 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.760956 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.760989 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.761428 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.761429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.761435 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:19.761824 (+ 389us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:19.761825 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:19.761839 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:19.761859 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:19.762132 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:19.762138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:19.762141 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.762203 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.762217 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":68,"threads_started":1}
I20260620 14:51:19.762475 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.760456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.5.118.131:46315)
0620 14:51:19.760683 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.760686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.760855 (+ 169us) server_negotiation.cc:407] Connection header received
0620 14:51:19.760911 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.760915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.760927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.761124 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:19.762098 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:19.762105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:19.762106 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:19.762110 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:19.762153 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:19.762173 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:19.762251 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.762369 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.762391 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
I20260620 14:51:19.762733 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.760743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.5.118.129:39181)
0620 14:51:19.761004 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.761009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.761022 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:51:19.761049 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.761051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.761059 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.761126 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:19.761945 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:19.761956 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:19.761957 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:19.761961 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:19.761994 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:19.762046 (+ 52us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:19.762106 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.762642 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.762659 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
I20260620 14:51:19.762758 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.760428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.5.118.130:34375)
0620 14:51:19.760607 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.760613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.760854 (+ 241us) server_negotiation.cc:407] Connection header received
0620 14:51:19.760960 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.760964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.760973 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.761040 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:19.762424 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:19.762433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:19.762433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:19.762438 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:19.762474 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:19.762499 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:19.762557 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.762654 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.762670 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
I20260620 14:51:19.762447 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.760350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:46315 (local address 127.0.0.1:51556)
0620 14:51:19.760770 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.760783 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.760852 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.761425 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.761426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.761432 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:19.762008 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:19.762009 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:19.762017 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:19.762032 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:19.762297 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:19.762298 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:19.762299 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.762351 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.762364 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":92,"threads_started":1}
I20260620 14:51:19.762686 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.760313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:34375 (local address 127.0.0.1:46446)
0620 14:51:19.760738 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.760770 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.760880 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.761419 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.761424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.761432 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:19.762328 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:19.762329 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:19.762336 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:19.762347 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:19.762551 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:19.762552 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:19.762553 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.762610 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.762622 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":76,"threads_started":1}
I20260620 14:51:19.763650 10468 tablet_service.cc:1467] Processing CreateTablet for tablet 5f33938fc8f6421c95e70d91c82a9409 (DEFAULT_TABLE table=test-table [id=6cc03fd195e34de59655d93b7334d6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:19.763645 10599 tablet_service.cc:1467] Processing CreateTablet for tablet 5f33938fc8f6421c95e70d91c82a9409 (DEFAULT_TABLE table=test-table [id=6cc03fd195e34de59655d93b7334d6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:19.763643 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 5f33938fc8f6421c95e70d91c82a9409 (DEFAULT_TABLE table=test-table [id=6cc03fd195e34de59655d93b7334d6c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:19.764010 10468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f33938fc8f6421c95e70d91c82a9409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:19.764010 10599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f33938fc8f6421c95e70d91c82a9409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:19.764010 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f33938fc8f6421c95e70d91c82a9409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:19.766636 10679 tablet_bootstrap.cc:492] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135: Bootstrap starting.
I20260620 14:51:19.767403 10679 tablet_bootstrap.cc:654] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:19.767459 10680 tablet_bootstrap.cc:492] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9: Bootstrap starting.
I20260620 14:51:19.767800 10679 log.cc:824] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:19.768111 10681 tablet_bootstrap.cc:492] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46: Bootstrap starting.
I20260620 14:51:19.768261 10680 tablet_bootstrap.cc:654] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:19.768585 10680 log.cc:824] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:19.768746 10679 tablet_bootstrap.cc:492] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135: No bootstrap required, opened a new log
I20260620 14:51:19.768812 10679 ts_tablet_manager.cc:1397] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:51:19.769280 10681 tablet_bootstrap.cc:654] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:19.769335 10680 tablet_bootstrap.cc:492] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9: No bootstrap required, opened a new log
I20260620 14:51:19.769403 10680 ts_tablet_manager.cc:1397] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:51:19.769841 10681 log.cc:824] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:19.771453 10679 raft_consensus.cc:348] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34375 } } peers { permanent_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46315 } } peers { permanent_uuid: "0b920a3821ab4d9e85599449f064b7e9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39181 } }
I20260620 14:51:19.771589 10679 raft_consensus.cc:374] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:19.771611 10679 raft_consensus.cc:729] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc1fd856a984c0e8c2b629bc8cd6135, State: Initialized, Role: FOLLOWER
I20260620 14:51:19.771739 10679 consensus_queue.cc:260] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34375 } } peers { permanent_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46315 } } peers { permanent_uuid: "0b920a3821ab4d9e85599449f064b7e9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39181 } }
I20260620 14:51:19.772004 10679 ts_tablet_manager.cc:1428] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:19.772013 10534 heartbeater.cc:503] Master 127.5.118.190:40361 was elected leader, sending a full tablet report...
I20260620 14:51:19.772002 10680 raft_consensus.cc:348] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34375 } } peers { permanent_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46315 } } peers { permanent_uuid: "0b920a3821ab4d9e85599449f064b7e9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39181 } }
I20260620 14:51:19.772207 10680 raft_consensus.cc:374] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:19.772256 10680 raft_consensus.cc:729] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b920a3821ab4d9e85599449f064b7e9, State: Initialized, Role: FOLLOWER
I20260620 14:51:19.772387 10680 consensus_queue.cc:260] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34375 } } peers { permanent_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46315 } } peers { permanent_uuid: "0b920a3821ab4d9e85599449f064b7e9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39181 } }
I20260620 14:51:19.772639 10680 ts_tablet_manager.cc:1428] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 14:51:19.772647 10403 heartbeater.cc:503] Master 127.5.118.190:40361 was elected leader, sending a full tablet report...
I20260620 14:51:19.772843 10681 tablet_bootstrap.cc:492] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46: No bootstrap required, opened a new log
I20260620 14:51:19.772909 10681 ts_tablet_manager.cc:1397] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 14:51:19.775121 10681 raft_consensus.cc:348] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34375 } } peers { permanent_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46315 } } peers { permanent_uuid: "0b920a3821ab4d9e85599449f064b7e9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39181 } }
I20260620 14:51:19.775283 10681 raft_consensus.cc:374] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:19.775329 10681 raft_consensus.cc:729] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7b40e2f6bf45589dd5d21351ea8d46, State: Initialized, Role: FOLLOWER
I20260620 14:51:19.775439 10681 consensus_queue.cc:260] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34375 } } peers { permanent_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46315 } } peers { permanent_uuid: "0b920a3821ab4d9e85599449f064b7e9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39181 } }
I20260620 14:51:19.775688 10681 ts_tablet_manager.cc:1428] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:51:19.775776 10665 heartbeater.cc:503] Master 127.5.118.190:40361 was elected leader, sending a full tablet report...
W20260620 14:51:19.866974 10535 tablet.cc:2369] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:19.907490 10686 raft_consensus.cc:482] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:19.907637 10686 raft_consensus.cc:504] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34375 } } peers { permanent_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46315 } } peers { permanent_uuid: "0b920a3821ab4d9e85599449f064b7e9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39181 } }
I20260620 14:51:19.908248 10686 leader_election.cc:288] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bc1fd856a984c0e8c2b629bc8cd6135 (127.5.118.130:34375), 6c7b40e2f6bf45589dd5d21351ea8d46 (127.5.118.131:46315)
I20260620 14:51:19.909585 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.908505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:39143 (local address 127.5.118.131:46315)
0620 14:51:19.908560 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.908562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.908725 (+ 163us) server_negotiation.cc:407] Connection header received
0620 14:51:19.908755 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.908756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.908761 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.908847 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:19.909348 (+ 501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:19.909349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:19.909349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:19.909354 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:19.909368 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:19.909373 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:19.909429 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.909504 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.909516 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:51:19.909585 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.908500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:41949 (local address 127.5.118.130:34375)
0620 14:51:19.908548 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.908550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.908560 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:19.908588 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.908589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.908594 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.908686 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:19.909279 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:19.909280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:19.909280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:19.909284 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:19.909323 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:19.909329 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:19.909398 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.909494 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.909510 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:51:19.909590 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.908403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:34375 (local address 127.5.118.129:41949)
0620 14:51:19.908455 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.908468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.908507 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.908708 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.908709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.908712 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:19.909129 (+ 417us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:19.909130 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:19.909136 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:19.909196 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:19.909434 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:19.909435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:19.909435 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.909490 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.909511 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":3}
I20260620 14:51:19.909624 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.908407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:46315 (local address 127.5.118.129:39143)
0620 14:51:19.908625 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:19.908633 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:19.908701 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:19.908855 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:19.908856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:19.908858 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:19.909250 (+ 392us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:19.909251 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:19.909256 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:19.909266 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:19.909434 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:19.909435 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:19.909435 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 14:51:19.909491 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:51:19.909508 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":44,"thread_start_us":102,"threads_started":1}
I20260620 14:51:19.909997 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f33938fc8f6421c95e70d91c82a9409" candidate_uuid: "0b920a3821ab4d9e85599449f064b7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" is_pre_election: true
I20260620 14:51:19.910171 10619 raft_consensus.cc:2454] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b920a3821ab4d9e85599449f064b7e9 in term 0.
I20260620 14:51:19.910184 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f33938fc8f6421c95e70d91c82a9409" candidate_uuid: "0b920a3821ab4d9e85599449f064b7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" is_pre_election: true
I20260620 14:51:19.910351 10488 raft_consensus.cc:2454] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b920a3821ab4d9e85599449f064b7e9 in term 0.
I20260620 14:51:19.910420 10293 leader_election.cc:302] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b920a3821ab4d9e85599449f064b7e9, 6c7b40e2f6bf45589dd5d21351ea8d46; no voters:
I20260620 14:51:19.910563 10686 raft_consensus.cc:2781] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:19.910666 10686 raft_consensus.cc:482] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:19.910699 10686 raft_consensus.cc:3037] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:19.978387 10686 raft_consensus.cc:504] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34375 } } peers { permanent_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46315 } } peers { permanent_uuid: "0b920a3821ab4d9e85599449f064b7e9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39181 } }
I20260620 14:51:19.978611 10686 leader_election.cc:288] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [CANDIDATE]: Term 1 election: Requested vote from peers 5bc1fd856a984c0e8c2b629bc8cd6135 (127.5.118.130:34375), 6c7b40e2f6bf45589dd5d21351ea8d46 (127.5.118.131:46315)
I20260620 14:51:19.978886 10619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f33938fc8f6421c95e70d91c82a9409" candidate_uuid: "0b920a3821ab4d9e85599449f064b7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46"
I20260620 14:51:19.978886 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f33938fc8f6421c95e70d91c82a9409" candidate_uuid: "0b920a3821ab4d9e85599449f064b7e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135"
I20260620 14:51:19.978989 10619 raft_consensus.cc:3037] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:19.978988 10488 raft_consensus.cc:3037] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:19.979974 10619 raft_consensus.cc:2454] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b920a3821ab4d9e85599449f064b7e9 in term 1.
I20260620 14:51:19.979974 10488 raft_consensus.cc:2454] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b920a3821ab4d9e85599449f064b7e9 in term 1.
I20260620 14:51:19.980219 10290 leader_election.cc:302] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b920a3821ab4d9e85599449f064b7e9, 5bc1fd856a984c0e8c2b629bc8cd6135; no voters:
I20260620 14:51:19.980336 10686 raft_consensus.cc:2781] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:19.980463 10686 raft_consensus.cc:686] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [term 1 LEADER]: Becoming Leader. State: Replica: 0b920a3821ab4d9e85599449f064b7e9, State: Running, Role: LEADER
I20260620 14:51:19.980578 10686 consensus_queue.cc:237] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34375 } } peers { permanent_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46315 } } peers { permanent_uuid: "0b920a3821ab4d9e85599449f064b7e9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39181 } }
I20260620 14:51:19.981520 10215 catalog_manager.cc:5495] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b920a3821ab4d9e85599449f064b7e9 (127.5.118.129). New cstate: current_term: 1 leader_uuid: "0b920a3821ab4d9e85599449f064b7e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b920a3821ab4d9e85599449f064b7e9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39181 } health_report { overall_health: HEALTHY } } }
W20260620 14:51:19.987176 10404 tablet.cc:2369] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:51:19.988029 10666 tablet.cc:2369] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:19.992502 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:19.991494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.5.118.129:39181)
0620 14:51:19.991535 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:19.991537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:19.991551 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:51:19.991607 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:19.991608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:19.991619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:19.991674 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:19.992272 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:19.992273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:19.992274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:19.992278 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:19.992292 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:19.992295 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:19.992344 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:19.992389 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 14:51:19.992400 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:51:19.994278 10488 raft_consensus.cc:1261] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135 [term 1 FOLLOWER]: Refusing update from remote peer 0b920a3821ab4d9e85599449f064b7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:19.994598 10619 raft_consensus.cc:1261] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46 [term 1 FOLLOWER]: Refusing update from remote peer 0b920a3821ab4d9e85599449f064b7e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:19.994635 10689 consensus_queue.cc:1035] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bc1fd856a984c0e8c2b629bc8cd6135" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:19.994890 10689 consensus_queue.cc:1035] T 5f33938fc8f6421c95e70d91c82a9409 P 0b920a3821ab4d9e85599449f064b7e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7b40e2f6bf45589dd5d21351ea8d46" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 46315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:19.997327 10693 mvcc.cc:205] Tried to move back new op lower bound from 7298937159653806080 to 7298937159601098752. Current Snapshot: MvccSnapshot[applied={T|T < 7298937159653806080}]
I20260620 14:51:19.997480 10694 mvcc.cc:205] Tried to move back new op lower bound from 7298937159653806080 to 7298937159601098752. Current Snapshot: MvccSnapshot[applied={T|T < 7298937159653806080}]
I20260620 14:51:19.999610 10215 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:19.999719 10215 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:20.000552 10215 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c01b2a057dc43b988062a86b016486f: Sending DeleteTablet for 3 replicas of tablet 5f33938fc8f6421c95e70d91c82a9409
I20260620 14:51:20.000950 10468 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f33938fc8f6421c95e70d91c82a9409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:19 UTC) from {username='slave'} at 127.0.0.1:46446
I20260620 14:51:20.000967 10599 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f33938fc8f6421c95e70d91c82a9409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:19 UTC) from {username='slave'} at 127.0.0.1:51556
I20260620 14:51:20.000974 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f33938fc8f6421c95e70d91c82a9409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:19 UTC) from {username='slave'} at 127.0.0.1:35282
I20260620 14:51:20.001199 10704 tablet_replica.cc:330] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46: stopping tablet replica
I20260620 14:51:20.001292 10704 raft_consensus.cc:2229] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:20.001397 10704 raft_consensus.cc:2258] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:20.001456 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 10275
I20260620 14:51:20.002532 10702 tablet_replica.cc:330] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135: stopping tablet replica
I20260620 14:51:20.002656 10702 raft_consensus.cc:2229] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:20.002765 10702 raft_consensus.cc:2258] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:20.002866 10704 ts_tablet_manager.cc:1905] T 5f33938fc8f6421c95e70d91c82a9409 P 6c7b40e2f6bf45589dd5d21351ea8d46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:20.003190 10702 ts_tablet_manager.cc:1905] T 5f33938fc8f6421c95e70d91c82a9409 P 5bc1fd856a984c0e8c2b629bc8cd6135: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:51:20.007493 10199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:39181 (error 108)
I20260620 14:51:20.008155 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:20.007918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39181 (local address 127.0.0.1:35298)
0620 14:51:20.008027 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:20.008111 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 14:51:20.008234 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 10406
W20260620 14:51:20.008358 10199 catalog_manager.cc:4584] TS 0b920a3821ab4d9e85599449f064b7e9 (127.5.118.129:39181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f33938fc8f6421c95e70d91c82a9409: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39181: connect: Connection refused (error 111)
I20260620 14:51:20.061293 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:20.061107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39181 (local address 127.0.0.1:35308)
0620 14:51:20.061200 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:20.061252 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:51:20.061853 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:20.061757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39181 (local address 127.0.0.1:35316)
0620 14:51:20.061802 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:20.061830 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 14:51:20.079121 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 10537
I20260620 14:51:20.079202 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:20.079046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:34375 (local address 127.0.0.1:46450)
0620 14:51:20.079113 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:20.079157 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.130:34375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:51:20.087577 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 10184
I20260620 14:51:20.087754 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:20.087540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:46315 (local address 127.0.0.1:51570)
0620 14:51:20.087664 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:20.087702 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.131:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
2026-06-20T14:51:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (888 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:51:20.107816 5594 mini_oidc.cc:152] Starting JWKS server
I20260620 14:51:20.109144 5594 webserver.cc:460] Webserver started at https://127.0.0.1:39065/ using document root <none> and password file <none>
I20260620 14:51:20.109258 5594 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:51:20.109432 5594 webserver.cc:460] Webserver started at http://127.0.0.1:39959/ using document root <none> and password file <none>
2026-06-20T14:51:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:20Z Disabled control of system clock
I20260620 14:51:20.117516 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:39819
--webserver_interface=127.5.118.190
--webserver_port=0
--builtin_ntp_servers=127.5.118.148:35173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:39819
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39065/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/testchainca.cert with env {}
W20260620 14:51:20.203334 10711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:20.203553 10711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:20.203576 10711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:20.205260 10711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:20.205314 10711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:20.205328 10711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:20.205341 10711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:20.205353 10711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:51:20.205366 10711 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:51:20.205417 10711 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39065/jwks.json
I20260620 14:51:20.207075 10711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:35173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:39819
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:39819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39065/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:20.207371 10711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:20.207612 10711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:20.212833 10717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:20.212973 10711 server_base.cc:1034] running on GCE node
W20260620 14:51:20.212845 10719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:20.212786 10716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:20.213452 10711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:20.213802 10711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:20.214972 10711 hybrid_clock.cc:648] HybridClock initialized: now 1781967080214950 us; error 40 us; skew 500 ppm
I20260620 14:51:20.216338 10711 webserver.cc:460] Webserver started at http://127.5.118.190:36977/ using document root <none> and password file <none>
I20260620 14:51:20.216612 10711 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:20.216755 10711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:20.216914 10711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:20.218084 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "eecababaf1f946208bfcf5e3fb620a22"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "ee9a0bac07328c5bf9c0a0b6ef1f0053"
server_key_iv: "028605357cc3d0e19d1d3f22fc6fe08a"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.218487 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "eecababaf1f946208bfcf5e3fb620a22"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "ee9a0bac07328c5bf9c0a0b6ef1f0053"
server_key_iv: "028605357cc3d0e19d1d3f22fc6fe08a"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.219893 10711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:20.221541 10725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:20.221779 10711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:20.221879 10711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "eecababaf1f946208bfcf5e3fb620a22"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "ee9a0bac07328c5bf9c0a0b6ef1f0053"
server_key_iv: "028605357cc3d0e19d1d3f22fc6fe08a"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.221964 10711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:20.233165 10711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:20.233502 10711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:20.233668 10711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:20.237689 10711 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:39819
I20260620 14:51:20.237748 10777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:39819 every 8 connection(s)
I20260620 14:51:20.238123 10711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:20.239143 10778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:20.241633 10778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22: Bootstrap starting.
I20260620 14:51:20.242269 10778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:20.242578 10778 log.cc:824] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:20.243037 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 10711
I20260620 14:51:20.243184 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:20.243359 5594 external_mini_cluster.cc:1428] Setting key c4b021862d18a671d3ea8a9cc5352a79
I20260620 14:51:20.243465 10778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22: No bootstrap required, opened a new log
I20260620 14:51:20.246515 10778 raft_consensus.cc:348] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecababaf1f946208bfcf5e3fb620a22" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 39819 } }
I20260620 14:51:20.246714 10778 raft_consensus.cc:374] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:20.246747 10778 raft_consensus.cc:729] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eecababaf1f946208bfcf5e3fb620a22, State: Initialized, Role: FOLLOWER
I20260620 14:51:20.246888 10778 consensus_queue.cc:260] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecababaf1f946208bfcf5e3fb620a22" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 39819 } }
I20260620 14:51:20.246986 10778 raft_consensus.cc:388] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:20.247033 10778 raft_consensus.cc:482] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:20.247084 10778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:20.249720 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:20.243902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.5.118.190:39819)
0620 14:51:20.244152 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:20.244162 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:20.244224 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:51:20.245003 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:20.245013 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:20.245096 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:20.245269 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:20.245433 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.246821 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.247836 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.248358 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.248454 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:20.249076 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:20.249080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:20.249080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:20.249327 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:20.249363 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:20.249376 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:20.249430 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:20.249492 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:51:20.249559 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260620 14:51:20.250677 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:51:20.268277 10778 raft_consensus.cc:504] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecababaf1f946208bfcf5e3fb620a22" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 39819 } }
I20260620 14:51:20.268482 10778 leader_election.cc:302] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eecababaf1f946208bfcf5e3fb620a22; no voters:
I20260620 14:51:20.268738 10778 leader_election.cc:288] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:20.268831 10783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:20.269042 10783 raft_consensus.cc:686] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [term 1 LEADER]: Becoming Leader. State: Replica: eecababaf1f946208bfcf5e3fb620a22, State: Running, Role: LEADER
I20260620 14:51:20.269065 10778 sys_catalog.cc:564] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:20.269219 10783 consensus_queue.cc:237] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecababaf1f946208bfcf5e3fb620a22" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 39819 } }
I20260620 14:51:20.269647 10785 sys_catalog.cc:455] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eecababaf1f946208bfcf5e3fb620a22. Latest consensus state: current_term: 1 leader_uuid: "eecababaf1f946208bfcf5e3fb620a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecababaf1f946208bfcf5e3fb620a22" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 39819 } } }
I20260620 14:51:20.269737 10785 sys_catalog.cc:458] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:20.269981 10784 sys_catalog.cc:455] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eecababaf1f946208bfcf5e3fb620a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eecababaf1f946208bfcf5e3fb620a22" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 39819 } } }
I20260620 14:51:20.270326 10789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:20.270444 10784 sys_catalog.cc:458] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:20.270942 10789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:20.272899 10789 catalog_manager.cc:1347] Generated new cluster ID: acc3b355f3ab4aaa9167f277875a1bac
I20260620 14:51:20.272964 10789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:20.277593 10789 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:20.278119 10789 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:20.286396 10789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eecababaf1f946208bfcf5e3fb620a22: Generated new TSK 0
I20260620 14:51:20.286633 10789 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:51:20.319468 10808 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:51:20.319682 10808 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:51:20.319793 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:20.295833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.5.118.190:39819)
0620 14:51:20.295880 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:20.295883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:20.296044 (+ 161us) server_negotiation.cc:407] Connection header received
0620 14:51:20.296159 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:20.296160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:20.296189 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:20.296257 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:51:20.296419 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.296973 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.297777 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.297988 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.298082 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:20.298331 (+ 249us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:51:20.319466 (+ 21135us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:51:20.319691 (+ 225us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 14:51:20.319972 10782 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:51:20.320276 10807 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:20.295759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:39819 (local address 127.0.0.1:32982)
0620 14:51:20.295965 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:20.295976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:20.296068 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:20.296262 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:20.296263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:20.296265 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:51:20.296334 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:20.296337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.297078 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.297080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:20.297682 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:20.297684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.298073 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.298074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:20.298155 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:20.298181 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:51:20.319632 (+ 21451us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:51:20.319762 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.118.190:39819: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.118.190:39819: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39065/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:51:20.320847 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 10711
2026-06-20T14:51:20Z chronyd exiting
I20260620 14:51:20.356804 5594 test_util.cc:175] -----------------------------------------------
I20260620 14:51:20.356889 5594 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967053866185-5594-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (250 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:51:20.357743 5594 mini_oidc.cc:152] Starting JWKS server
I20260620 14:51:20.358464 5594 webserver.cc:460] Webserver started at https://127.0.0.1:42695/ using document root <none> and password file <none>
I20260620 14:51:20.358536 5594 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:51:20.358706 5594 webserver.cc:460] Webserver started at http://127.0.0.1:35211/ using document root <none> and password file <none>
2026-06-20T14:51:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:20Z Disabled control of system clock
I20260620 14:51:20.366679 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43639
--webserver_interface=127.5.118.190
--webserver_port=0
--builtin_ntp_servers=127.5.118.148:35677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:43639
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42695/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/testchainca.cert with env {}
W20260620 14:51:20.448712 10814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:20.448974 10814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:20.449014 10814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:20.450230 10814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:20.450309 10814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:20.450328 10814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:20.450348 10814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:20.450376 10814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:51:20.450407 10814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:51:20.450433 10814 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42695/jwks.json
I20260620 14:51:20.451993 10814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:35677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:43639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42695/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:20.452317 10814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:20.452548 10814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:20.457599 10819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:20.457582 10820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:20.457724 10822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:20.458081 10814 server_base.cc:1034] running on GCE node
I20260620 14:51:20.458355 10814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:20.458683 10814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:20.459884 10814 hybrid_clock.cc:648] HybridClock initialized: now 1781967080459849 us; error 57 us; skew 500 ppm
I20260620 14:51:20.461513 10814 webserver.cc:460] Webserver started at http://127.5.118.190:44735/ using document root <none> and password file <none>
I20260620 14:51:20.461817 10814 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:20.461897 10814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:20.462051 10814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:20.463284 10814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "e03624e834b14c3a9d4b944fb138376d"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "be2f46aed0947c78e0990a20ae806f12"
server_key_iv: "d8ef34f9d82ac08cc26e4c9c1c1609ed"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.463707 10814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "e03624e834b14c3a9d4b944fb138376d"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "be2f46aed0947c78e0990a20ae806f12"
server_key_iv: "d8ef34f9d82ac08cc26e4c9c1c1609ed"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.465344 10814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260620 14:51:20.466393 10828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:20.466646 10814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:20.466771 10814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "e03624e834b14c3a9d4b944fb138376d"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "be2f46aed0947c78e0990a20ae806f12"
server_key_iv: "d8ef34f9d82ac08cc26e4c9c1c1609ed"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.466856 10814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:20.477784 10814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:20.478165 10814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:20.478343 10814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:20.482883 10814 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:43639
I20260620 14:51:20.482915 10880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:43639 every 8 connection(s)
I20260620 14:51:20.483306 10814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:20.484441 10881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:20.486922 10881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d: Bootstrap starting.
I20260620 14:51:20.487488 10881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:20.487733 10881 log.cc:824] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:20.488531 10881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d: No bootstrap required, opened a new log
I20260620 14:51:20.490183 10881 raft_consensus.cc:348] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03624e834b14c3a9d4b944fb138376d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43639 } }
I20260620 14:51:20.490335 10881 raft_consensus.cc:374] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:20.490360 10881 raft_consensus.cc:729] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e03624e834b14c3a9d4b944fb138376d, State: Initialized, Role: FOLLOWER
I20260620 14:51:20.490535 10881 consensus_queue.cc:260] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03624e834b14c3a9d4b944fb138376d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43639 } }
I20260620 14:51:20.490619 10881 raft_consensus.cc:388] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:20.490649 10881 raft_consensus.cc:482] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:20.490679 10881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:20.491441 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 10814
I20260620 14:51:20.491528 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:20.491725 5594 external_mini_cluster.cc:1428] Setting key 94056c84fabe5652cab3200a84aa4538
I20260620 14:51:20.496861 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:20.492308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.5.118.190:43639)
0620 14:51:20.492515 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:20.492523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:20.492549 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:51:20.492649 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:20.492655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:20.492742 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:20.492858 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:20.493049 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.494013 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.494804 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.495288 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.495386 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:20.496040 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:20.496044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:20.496044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:20.496310 (+ 266us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:20.496348 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:20.496359 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:20.496415 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:20.496477 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 14:51:20.496542 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260620 14:51:20.497958 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:51:20.560624 10881 raft_consensus.cc:504] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03624e834b14c3a9d4b944fb138376d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43639 } }
I20260620 14:51:20.560797 10881 leader_election.cc:302] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e03624e834b14c3a9d4b944fb138376d; no voters:
I20260620 14:51:20.560992 10881 leader_election.cc:288] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:20.561091 10886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:20.561282 10881 sys_catalog.cc:564] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:20.561331 10886 raft_consensus.cc:686] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [term 1 LEADER]: Becoming Leader. State: Replica: e03624e834b14c3a9d4b944fb138376d, State: Running, Role: LEADER
I20260620 14:51:20.561461 10886 consensus_queue.cc:237] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03624e834b14c3a9d4b944fb138376d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43639 } }
I20260620 14:51:20.561901 10888 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e03624e834b14c3a9d4b944fb138376d. Latest consensus state: current_term: 1 leader_uuid: "e03624e834b14c3a9d4b944fb138376d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03624e834b14c3a9d4b944fb138376d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43639 } } }
I20260620 14:51:20.561992 10888 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:20.562140 10887 sys_catalog.cc:455] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e03624e834b14c3a9d4b944fb138376d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e03624e834b14c3a9d4b944fb138376d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43639 } } }
I20260620 14:51:20.562265 10887 sys_catalog.cc:458] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:20.562668 10894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:20.563387 10894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:20.564986 10894 catalog_manager.cc:1347] Generated new cluster ID: 778059ac36e641248e4995113d921252
I20260620 14:51:20.565049 10894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:20.578485 10894 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:20.579051 10894 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:20.585614 10894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e03624e834b14c3a9d4b944fb138376d: Generated new TSK 0
I20260620 14:51:20.585888 10894 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:51:20.626329 10911 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:51:20.626430 10911 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:51:20.626694 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:20.604862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.5.118.190:43639)
0620 14:51:20.604910 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:20.604912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:20.605018 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:51:20.605097 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:20.605098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:20.605109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:20.605178 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:51:20.605376 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.605898 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.607084 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.607278 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.607333 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:20.607545 (+ 212us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:51:20.626455 (+ 18910us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:51:20.626633 (+ 178us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49104: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 14:51:20.626765 10885 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49104: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 14:51:20.626809 10910 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:20.604762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43639 (local address 127.0.0.1:49104)
0620 14:51:20.604936 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:20.604948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:20.605024 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:20.605200 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:20.605200 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:20.605202 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:51:20.605271 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:20.605274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.606046 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.606047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:20.606994 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:20.606997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.607365 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.607366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:20.607429 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:20.607435 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:51:20.626582 (+ 19147us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:51:20.626709 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.118.190:43639: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42695/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260620 14:51:20.627008 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 10814
2026-06-20T14:51:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (307 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:51:20.664920 5594 mini_oidc.cc:152] Starting JWKS server
I20260620 14:51:20.665642 5594 webserver.cc:460] Webserver started at https://127.0.0.1:43857/ using document root <none> and password file <none>
I20260620 14:51:20.665688 5594 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:51:20.665863 5594 webserver.cc:460] Webserver started at http://127.0.0.1:35505/ using document root <none> and password file <none>
2026-06-20T14:51:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:20Z Disabled control of system clock
I20260620 14:51:20.674288 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:41369
--webserver_interface=127.5.118.190
--webserver_port=0
--builtin_ntp_servers=127.5.118.148:46047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:41369
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43857/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:51:20.756397 10917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:20.756618 10917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:20.756640 10917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:20.757882 10917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:20.757931 10917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:20.757944 10917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:20.757957 10917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:20.758009 10917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:51:20.758028 10917 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:51:20.758049 10917 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43857/jwks.json
I20260620 14:51:20.759521 10917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:46047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:41369
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:41369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43857/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:20.759794 10917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:20.760115 10917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:20.765192 10923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:20.765239 10925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:20.765538 10917 server_base.cc:1034] running on GCE node
W20260620 14:51:20.765199 10922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:20.765830 10917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:20.766157 10917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:20.767304 10917 hybrid_clock.cc:648] HybridClock initialized: now 1781967080767263 us; error 62 us; skew 500 ppm
I20260620 14:51:20.768808 10917 webserver.cc:460] Webserver started at http://127.5.118.190:43943/ using document root <none> and password file <none>
I20260620 14:51:20.769069 10917 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:20.769138 10917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:20.769315 10917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:20.770430 10917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "cc7c1f09623f4a9587dd13130d0b824b"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "887b4c57af26f61f773f8ea844e0f1ac"
server_key_iv: "d718b4d2d517f78716f3b98f3a70845a"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.770884 10917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "cc7c1f09623f4a9587dd13130d0b824b"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "887b4c57af26f61f773f8ea844e0f1ac"
server_key_iv: "d718b4d2d517f78716f3b98f3a70845a"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.773252 10917 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 14:51:20.779336 10931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:20.779654 10917 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260620 14:51:20.779770 10917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "cc7c1f09623f4a9587dd13130d0b824b"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "887b4c57af26f61f773f8ea844e0f1ac"
server_key_iv: "d718b4d2d517f78716f3b98f3a70845a"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.779873 10917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:20.793993 10917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:20.794296 10917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:20.794411 10917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:20.798725 10917 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:41369
I20260620 14:51:20.798759 10983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:41369 every 8 connection(s)
I20260620 14:51:20.799155 10917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:20.799875 10984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:20.801358 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 10917
I20260620 14:51:20.801438 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:20.801643 5594 external_mini_cluster.cc:1428] Setting key a251667d850cdc355d15a4826ecadb86
I20260620 14:51:20.803252 10984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b: Bootstrap starting.
I20260620 14:51:20.803952 10984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:20.804308 10984 log.cc:824] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:20.805018 10984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b: No bootstrap required, opened a new log
I20260620 14:51:20.806747 10984 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7c1f09623f4a9587dd13130d0b824b" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41369 } }
I20260620 14:51:20.806905 10984 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:20.806950 10984 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7c1f09623f4a9587dd13130d0b824b, State: Initialized, Role: FOLLOWER
I20260620 14:51:20.807112 10984 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7c1f09623f4a9587dd13130d0b824b" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41369 } }
I20260620 14:51:20.807238 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:20.802239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.5.118.190:41369)
0620 14:51:20.802472 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:20.802480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:20.802505 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:51:20.802924 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:20.802930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:20.802992 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:20.803167 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:20.803393 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.804451 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.805310 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.805738 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.805880 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:20.806609 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:20.806614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:20.806614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:20.806788 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:20.806821 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:20.806831 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:20.806907 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:20.806984 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:51:20.807067 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260620 14:51:20.807240 10984 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:20.807349 10984 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:20.807404 10984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:20.808084 10984 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7c1f09623f4a9587dd13130d0b824b" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41369 } }
I20260620 14:51:20.808218 10984 leader_election.cc:302] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc7c1f09623f4a9587dd13130d0b824b; no voters:
I20260620 14:51:20.808275 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:51:20.808398 10984 leader_election.cc:288] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:20.808421 10989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:20.808537 10989 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [term 1 LEADER]: Becoming Leader. State: Replica: cc7c1f09623f4a9587dd13130d0b824b, State: Running, Role: LEADER
I20260620 14:51:20.808621 10989 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7c1f09623f4a9587dd13130d0b824b" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41369 } }
I20260620 14:51:20.808669 10984 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:20.808939 10990 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc7c1f09623f4a9587dd13130d0b824b. Latest consensus state: current_term: 1 leader_uuid: "cc7c1f09623f4a9587dd13130d0b824b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7c1f09623f4a9587dd13130d0b824b" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41369 } } }
I20260620 14:51:20.809058 10990 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:20.808928 10991 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc7c1f09623f4a9587dd13130d0b824b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc7c1f09623f4a9587dd13130d0b824b" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41369 } } }
I20260620 14:51:20.809408 10991 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:20.809566 10995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:20.810034 10995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:20.813225 10995 catalog_manager.cc:1347] Generated new cluster ID: 1f2071876625434eaa9bb77a184a546e
I20260620 14:51:20.813293 10995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:20.823760 10995 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:20.824417 10995 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:20.833704 10995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc7c1f09623f4a9587dd13130d0b824b: Generated new TSK 0
I20260620 14:51:20.833935 10995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:51:20.844777 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:20.840774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.5.118.190:41369)
0620 14:51:20.840850 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:20.840853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:20.841048 (+ 195us) server_negotiation.cc:407] Connection header received
0620 14:51:20.841091 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:20.841092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:20.841110 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:20.841185 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:51:20.841387 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.841907 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.843052 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.843249 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.843307 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:20.843525 (+ 218us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:51:20.844529 (+ 1004us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:51:20.844700 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36790: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 14:51:20.844833 11013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:20.840701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:41369 (local address 127.0.0.1:36790)
0620 14:51:20.840901 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:20.840913 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:20.840992 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:20.841203 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:20.841204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:20.841207 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:51:20.841271 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:20.841278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.842001 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.842002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:20.842953 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:20.842957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:20.843334 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:20.843335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:20.843395 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:20.843401 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:51:20.844644 (+ 1243us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:51:20.844738 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.118.190:41369: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":120,"threads_started":1}
W20260620 14:51:20.844851 10987 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36790: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43857/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260620 14:51:20.845474 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 10917
E20260620 14:51:20.855924 11014 webserver.cc:533] Webserver: sslize failed:
E20260620 14:51:20.856009 11014 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T14:51:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:51:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:20Z Disabled control of system clock
I20260620 14:51:20.890497 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42549
--webserver_interface=127.5.118.190
--webserver_port=0
--builtin_ntp_servers=127.5.118.148:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:42549
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/testchainca.cert with env {}
W20260620 14:51:20.975131 11018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:20.975363 11018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:20.975389 11018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:20.976871 11018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:20.976931 11018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:20.976945 11018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:20.976964 11018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:20.976976 11018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:51:20.976990 11018 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:51:20.977003 11018 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:51:20.978621 11018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:42549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:20.978830 11018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:20.979018 11018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:20.984025 11026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:20.984036 11024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:20.984036 11023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:20.984086 11018 server_base.cc:1034] running on GCE node
I20260620 14:51:20.984543 11018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:20.984860 11018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:20.986027 11018 hybrid_clock.cc:648] HybridClock initialized: now 1781967080986022 us; error 40 us; skew 500 ppm
I20260620 14:51:20.987370 11018 webserver.cc:460] Webserver started at http://127.5.118.190:41613/ using document root <none> and password file <none>
I20260620 14:51:20.987620 11018 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:20.987702 11018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:20.987854 11018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:20.989002 11018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "b4ef1f10dcb840b8b2fef031c0b42609"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "b48b3d1459345c826231a9228a6355d4"
server_key_iv: "207ac99f4f4ec8142b1dd8a7b3c987ce"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.989379 11018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "b4ef1f10dcb840b8b2fef031c0b42609"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "b48b3d1459345c826231a9228a6355d4"
server_key_iv: "207ac99f4f4ec8142b1dd8a7b3c987ce"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.990731 11018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:51:20.991542 11032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:20.991868 11018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:20.991974 11018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "b4ef1f10dcb840b8b2fef031c0b42609"
format_stamp: "Formatted at 2026-06-20 14:51:20 on dist-test-slave-2m5f"
server_key: "b48b3d1459345c826231a9228a6355d4"
server_key_iv: "207ac99f4f4ec8142b1dd8a7b3c987ce"
server_key_version: "encryptionkey@0"
I20260620 14:51:20.992053 11018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:21.006211 11018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:21.006577 11018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:21.006752 11018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:21.010949 11018 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:42549
I20260620 14:51:21.010985 11084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:42549 every 8 connection(s)
I20260620 14:51:21.011354 11018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:21.012432 11085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:21.014818 11085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609: Bootstrap starting.
I20260620 14:51:21.015144 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 11018
I20260620 14:51:21.015249 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:21.015440 5594 external_mini_cluster.cc:1428] Setting key 9ea1173e731e76a8481b8308a0497ffe
I20260620 14:51:21.015588 11085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:21.015944 11085 log.cc:824] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:21.016799 11085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609: No bootstrap required, opened a new log
I20260620 14:51:21.019287 11085 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ef1f10dcb840b8b2fef031c0b42609" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42549 } }
I20260620 14:51:21.019439 11085 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:21.019462 11085 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ef1f10dcb840b8b2fef031c0b42609, State: Initialized, Role: FOLLOWER
I20260620 14:51:21.019610 11085 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ef1f10dcb840b8b2fef031c0b42609" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42549 } }
I20260620 14:51:21.019681 11085 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:21.019704 11085 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:21.019773 11085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:21.020601 11085 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ef1f10dcb840b8b2fef031c0b42609" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42549 } }
I20260620 14:51:21.020731 11085 leader_election.cc:302] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4ef1f10dcb840b8b2fef031c0b42609; no voters:
I20260620 14:51:21.020906 11085 leader_election.cc:288] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:21.021018 11090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:21.021267 11085 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:21.021513 11090 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [term 1 LEADER]: Becoming Leader. State: Replica: b4ef1f10dcb840b8b2fef031c0b42609, State: Running, Role: LEADER
I20260620 14:51:21.021641 11090 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ef1f10dcb840b8b2fef031c0b42609" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42549 } }
I20260620 14:51:21.022032 11090 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4ef1f10dcb840b8b2fef031c0b42609. Latest consensus state: current_term: 1 leader_uuid: "b4ef1f10dcb840b8b2fef031c0b42609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ef1f10dcb840b8b2fef031c0b42609" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42549 } } }
I20260620 14:51:21.022042 11091 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4ef1f10dcb840b8b2fef031c0b42609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ef1f10dcb840b8b2fef031c0b42609" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42549 } } }
I20260620 14:51:21.022163 11091 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:21.022359 11099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:21.022697 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:21.016053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.5.118.190:42549)
0620 14:51:21.016328 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:21.016344 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:21.016370 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:51:21.016975 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:21.016985 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:21.017063 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:21.017226 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:21.017447 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:21.018964 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:21.019985 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:21.020669 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:21.020801 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:21.021621 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:21.021625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:21.021626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:21.021837 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:21.021880 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:21.021899 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:21.021942 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:21.022379 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 14:51:21.022483 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":148,"threads_started":1}
I20260620 14:51:21.022116 11090 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:21.023255 11099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:21.023813 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:51:21.025277 11099 catalog_manager.cc:1347] Generated new cluster ID: fae9c70df7dc4e759e0f74d55f5f2181
I20260620 14:51:21.025347 11099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:21.033061 11099 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:21.033594 11099 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:21.049481 11099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4ef1f10dcb840b8b2fef031c0b42609: Generated new TSK 0
I20260620 14:51:21.049732 11099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:51:21.103571 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:21.055127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.5.118.190:42549)
0620 14:51:21.055180 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:21.055182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:21.055272 (+ 90us) server_negotiation.cc:407] Connection header received
0620 14:51:21.055361 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:21.055362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:21.055374 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:21.055448 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:51:21.055591 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:21.056127 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:21.056958 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:21.057176 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:21.057251 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:21.057451 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:51:21.103274 (+ 45823us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:51:21.103482 (+ 208us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43524: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 14:51:21.103660 11114 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:21.055021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42549 (local address 127.0.0.1:43524)
0620 14:51:21.055191 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:21.055202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:21.055292 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:21.055450 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:21.055451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:21.055453 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:51:21.055511 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:21.055513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:21.056254 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:21.056255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:21.056857 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:21.056859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:21.057255 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:21.057257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:21.057316 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:21.057321 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:51:21.103427 (+ 46106us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:51:21.103546 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.118.190:42549: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
W20260620 14:51:21.103674 11089 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43524: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260620 14:51:21.103976 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 11018
2026-06-20T14:51:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (238 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:21 dist-test-slave-2m5f krb5kdc[11126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:21 dist-test-slave-2m5f krb5kdc[11126](info): set up 2 sockets
Jun 20 14:51:21 dist-test-slave-2m5f krb5kdc[11126](info): commencing operation
krb5kdc: starting...
W20260620 14:51:22.336763 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:22 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:51:22.455595 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:46607
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:46607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:22.538241 11139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:22.538524 11139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:22.538569 11139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:22.539834 11139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:22.539916 11139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:22.539939 11139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:22.539961 11139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:22.539989 11139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:22.541613 11139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:46607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:46607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:22.541923 11139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:22.542182 11139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:22.547180 11144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:22.547266 11145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:22.547179 11147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:22.547420 11139 server_base.cc:1034] running on GCE node
I20260620 14:51:22.547688 11139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:22.548000 11139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:22.549172 11139 hybrid_clock.cc:648] HybridClock initialized: now 1781967082549159 us; error 56 us; skew 500 ppm
Jun 20 14:51:22 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967082, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:22.551815 11139 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:51:22.552381 11139 webserver.cc:460] Webserver started at http://127.5.118.190:38337/ using document root <none> and password file <none>
I20260620 14:51:22.552647 11139 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:22.552718 11139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:22.552870 11139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:22.553968 11139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "214c6f704299409e98780ab9ccb44dca"
format_stamp: "Formatted at 2026-06-20 14:51:22 on dist-test-slave-2m5f"
server_key: "65229139fc21affcca6b36d257746d35"
server_key_iv: "010845f3259f8a5b5d2b92f61db3c123"
server_key_version: "encryptionkey@0"
I20260620 14:51:22.554355 11139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "214c6f704299409e98780ab9ccb44dca"
format_stamp: "Formatted at 2026-06-20 14:51:22 on dist-test-slave-2m5f"
server_key: "65229139fc21affcca6b36d257746d35"
server_key_iv: "010845f3259f8a5b5d2b92f61db3c123"
server_key_version: "encryptionkey@0"
I20260620 14:51:22.555861 11139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:22.556982 11154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:22.557181 11139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:22.557287 11139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "214c6f704299409e98780ab9ccb44dca"
format_stamp: "Formatted at 2026-06-20 14:51:22 on dist-test-slave-2m5f"
server_key: "65229139fc21affcca6b36d257746d35"
server_key_iv: "010845f3259f8a5b5d2b92f61db3c123"
server_key_version: "encryptionkey@0"
I20260620 14:51:22.557379 11139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:22.572800 11139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:22.574836 11139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:22.575032 11139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:22.579351 11139 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:46607
I20260620 14:51:22.579401 11206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:46607 every 8 connection(s)
I20260620 14:51:22.579774 11139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:22.580768 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 11139
I20260620 14:51:22.580811 11207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:22.580881 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:22.581058 5594 external_mini_cluster.cc:1428] Setting key 4f08bb13d60b85d6e0411cf87d5e471f
I20260620 14:51:22.584035 11207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca: Bootstrap starting.
I20260620 14:51:22.584828 11207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:22.585136 11207 log.cc:824] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:22.585868 11207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca: No bootstrap required, opened a new log
Jun 20 14:51:22 dist-test-slave-2m5f krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:22.588507 11207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214c6f704299409e98780ab9ccb44dca" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46607 } }
I20260620 14:51:22.588657 11207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:22.588680 11207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214c6f704299409e98780ab9ccb44dca, State: Initialized, Role: FOLLOWER
I20260620 14:51:22.588835 11207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214c6f704299409e98780ab9ccb44dca" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46607 } }
I20260620 14:51:22.588896 11207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:22.588979 11207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:22.589057 11207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:22.589869 11207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214c6f704299409e98780ab9ccb44dca" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46607 } }
I20260620 14:51:22.589988 11207 leader_election.cc:302] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 214c6f704299409e98780ab9ccb44dca; no voters:
I20260620 14:51:22.590171 11207 leader_election.cc:288] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:22.590242 11212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:22.590438 11212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [term 1 LEADER]: Becoming Leader. State: Replica: 214c6f704299409e98780ab9ccb44dca, State: Running, Role: LEADER
I20260620 14:51:22.590451 11207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:22.591432 11212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214c6f704299409e98780ab9ccb44dca" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46607 } }
I20260620 14:51:22.592072 11213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "214c6f704299409e98780ab9ccb44dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214c6f704299409e98780ab9ccb44dca" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46607 } } }
I20260620 14:51:22.592401 11213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [sys.catalog]: This master's current role is: LEADER
W20260620 14:51:22.592509 11226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:51:22.592548 11226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:51:22.592080 11214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 214c6f704299409e98780ab9ccb44dca. Latest consensus state: current_term: 1 leader_uuid: "214c6f704299409e98780ab9ccb44dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214c6f704299409e98780ab9ccb44dca" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46607 } } }
I20260620 14:51:22.592880 11214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:22.592965 11228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:22.593107 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:22.581610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.5.118.190:46607)
0620 14:51:22.581792 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:22.581808 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:22.581834 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:51:22.582312 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:22.582318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:22.582367 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:22.582484 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:22.583306 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:22.584277 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:22.585346 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:22.585863 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:22.586002 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:22.588599 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:22.588604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:22.588604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:22.588618 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:22.590739 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:22.591182 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:22.591185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:22.591186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:22.591295 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:22.591554 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:22.591555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:22.591556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:22.591926 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:22.592575 (+ 649us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:22.592663 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 14:51:22.592818 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260620 14:51:22.593448 11228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:22.595099 11228 catalog_manager.cc:1347] Generated new cluster ID: 8b6b0473bd7c4810a760db5ee1e08932
I20260620 14:51:22.595171 11228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:22.608848 11228 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:22.609393 11228 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:22.613461 11228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 214c6f704299409e98780ab9ccb44dca: Generated new TSK 0
I20260620 14:51:22.613692 11228 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:51:22.644328 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46607
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:22.726842 11233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:22.727099 11233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:22.727142 11233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:22.728379 11233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:22.728452 11233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:22.728497 11233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:22.730065 11233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:22.730350 11233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:22.730612 11233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:22.736017 11239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:22.736019 11241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:22.736112 11238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:22.736166 11233 server_base.cc:1034] running on GCE node
I20260620 14:51:22.736629 11233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:22.736927 11233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:22.738124 11233 hybrid_clock.cc:648] HybridClock initialized: now 1781967082738108 us; error 51 us; skew 500 ppm
Jun 20 14:51:22 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967082, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:22.740851 11233 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:51:22.741350 11233 webserver.cc:460] Webserver started at http://127.5.118.129:45085/ using document root <none> and password file <none>
I20260620 14:51:22.741586 11233 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:22.741659 11233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:22.741815 11233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:22.742928 11233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "37c67cd130b948a4b3872877e11a7b20"
format_stamp: "Formatted at 2026-06-20 14:51:22 on dist-test-slave-2m5f"
server_key: "30942f9f45b8ae8a08e699b520c23de5"
server_key_iv: "6ea2cf6b4e29442a45211c78474d425f"
server_key_version: "encryptionkey@0"
I20260620 14:51:22.743315 11233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "37c67cd130b948a4b3872877e11a7b20"
format_stamp: "Formatted at 2026-06-20 14:51:22 on dist-test-slave-2m5f"
server_key: "30942f9f45b8ae8a08e699b520c23de5"
server_key_iv: "6ea2cf6b4e29442a45211c78474d425f"
server_key_version: "encryptionkey@0"
I20260620 14:51:22.744824 11233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:22.745656 11248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:22.745836 11233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:22.745944 11233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "37c67cd130b948a4b3872877e11a7b20"
format_stamp: "Formatted at 2026-06-20 14:51:22 on dist-test-slave-2m5f"
server_key: "30942f9f45b8ae8a08e699b520c23de5"
server_key_iv: "6ea2cf6b4e29442a45211c78474d425f"
server_key_version: "encryptionkey@0"
I20260620 14:51:22.746023 11233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:22.761858 11233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:22.763890 11233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:22.764068 11233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:22.764380 11233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:22.764711 11233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:22.764766 11233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:22.764803 11233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:22.764851 11233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:22.770677 11233 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:39903
I20260620 14:51:22.770740 11361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:39903 every 8 connection(s)
I20260620 14:51:22.771102 11233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:51:22 dist-test-slave-2m5f krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967082, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:22.779820 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 11233
I20260620 14:51:22.779956 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:22.780261 5594 external_mini_cluster.cc:1428] Setting key 1abe05b56f9284a022ccb39f0ae817cf
I20260620 14:51:22.780906 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:22.771916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:33575 (local address 127.5.118.190:46607)
0620 14:51:22.771967 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:22.771970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:22.772224 (+ 254us) server_negotiation.cc:407] Connection header received
0620 14:51:22.773171 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:22.773172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:22.773183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:22.773232 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:22.773826 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:22.774322 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:22.775317 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:22.775558 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:22.775632 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:22.777875 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:22.777876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:22.777876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:22.777884 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:22.779321 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:22.779761 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:22.779762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:22.779763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:22.779813 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:22.780103 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:22.780104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:22.780104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:22.780283 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:22.780370 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:22.780718 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 14:51:22.780805 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:51:22.781087 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:22.771907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:46607 (local address 127.5.118.129:33575)
0620 14:51:22.772078 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:22.772118 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:22.773082 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:22.773297 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:22.773301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:22.773552 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:22.773756 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:22.773760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:22.774399 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:22.774399 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:22.775224 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:22.775234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:22.775668 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:22.775669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:22.775773 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:22.776358 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:22.776366 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:22.777787 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:22.779398 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:22.779404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:22.779406 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:22.779684 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:22.779878 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:22.779879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:22.779880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:22.780030 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:22.780376 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:22.780383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:22.780553 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 14:51:22.780677 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 14:51:22.780806 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260620 14:51:22.781558 11362 heartbeater.cc:347] Connected to a master server at 127.5.118.190:46607
I20260620 14:51:22.781668 11362 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:22.781874 11362 heartbeater.cc:511] Master 127.5.118.190:46607 requested a full tablet report, sending...
I20260620 14:51:22.782351 11171 ts_manager.cc:194] Registered new tserver with Master: 37c67cd130b948a4b3872877e11a7b20 (127.5.118.129:39903)
I20260620 14:51:22.783003 11171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:33575
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:51:22.806983 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46607
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:22.888444 11367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:22.888670 11367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:22.888695 11367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:22.889845 11367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:22.889883 11367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:22.889909 11367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:22.891549 11367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:22.891753 11367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:22.891938 11367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:22.897588 11373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:22.897698 11372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:22.897596 11375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:22.898098 11367 server_base.cc:1034] running on GCE node
I20260620 14:51:22.898336 11367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:22.898555 11367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:22.899691 11367 hybrid_clock.cc:648] HybridClock initialized: now 1781967082899674 us; error 40 us; skew 500 ppm
Jun 20 14:51:22 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967082, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:22.902238 11367 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:51:22.902755 11367 webserver.cc:460] Webserver started at http://127.5.118.130:44045/ using document root <none> and password file <none>
I20260620 14:51:22.903004 11367 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:22.903051 11367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:22.903218 11367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:22.904392 11367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "22dd6c4747ee48588ea5d9788fb7810e"
format_stamp: "Formatted at 2026-06-20 14:51:22 on dist-test-slave-2m5f"
server_key: "d02caf16a41f210a2467a307d861fa82"
server_key_iv: "85af186cf2a32a650a16b828dcba0ea9"
server_key_version: "encryptionkey@0"
I20260620 14:51:22.904758 11367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "22dd6c4747ee48588ea5d9788fb7810e"
format_stamp: "Formatted at 2026-06-20 14:51:22 on dist-test-slave-2m5f"
server_key: "d02caf16a41f210a2467a307d861fa82"
server_key_iv: "85af186cf2a32a650a16b828dcba0ea9"
server_key_version: "encryptionkey@0"
I20260620 14:51:22.906113 11367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:22.906924 11382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:22.907147 11367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:22.907215 11367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "22dd6c4747ee48588ea5d9788fb7810e"
format_stamp: "Formatted at 2026-06-20 14:51:22 on dist-test-slave-2m5f"
server_key: "d02caf16a41f210a2467a307d861fa82"
server_key_iv: "85af186cf2a32a650a16b828dcba0ea9"
server_key_version: "encryptionkey@0"
I20260620 14:51:22.907310 11367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:22.921643 11367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:22.923601 11367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:22.923774 11367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:22.924019 11367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:22.924422 11367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:22.924476 11367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:22.924511 11367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:22.924538 11367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:22.930464 11367 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:41723
I20260620 14:51:22.930500 11495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:41723 every 8 connection(s)
I20260620 14:51:22.930877 11367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:22.931975 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 11367
I20260620 14:51:22.932085 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:22.932312 5594 external_mini_cluster.cc:1428] Setting key fa06853c8e350b200e4d892df24bd0a8
Jun 20 14:51:22 dist-test-slave-2m5f krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967082, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
I20260620 14:51:22.942402 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:22.931665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:53809 (local address 127.5.118.190:46607)
0620 14:51:22.931740 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:22.931743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:22.932037 (+ 294us) server_negotiation.cc:407] Connection header received
0620 14:51:22.933401 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:22.933402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:22.933412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:22.933455 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:22.934361 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:22.934887 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:22.936523 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:22.936719 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:22.936775 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:22.939295 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:22.939297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:22.939297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:22.939308 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:22.940801 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:22.941322 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:22.941324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:22.941324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:22.941386 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:22.941654 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:22.941656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:22.941656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:22.941777 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:22.941854 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:22.942216 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 14:51:22.942305 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:51:22.942618 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:22.931636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:46607 (local address 127.5.118.130:53809)
0620 14:51:22.931866 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:22.931927 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:22.933286 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:22.933551 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:22.933556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:22.933966 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:22.934272 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:22.934279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:22.934985 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:22.934986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:22.936434 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:22.936437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:22.936810 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:22.936811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:22.936968 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:22.937519 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:22.937531 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:22.939179 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:22.940912 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:22.940921 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:22.940923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:22.941228 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:22.941457 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:22.941458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:22.941458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:22.941582 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:22.941881 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:22.941888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:22.942070 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 14:51:22.942196 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:51:22.942365 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":108,"threads_started":1}
I20260620 14:51:22.943215 11496 heartbeater.cc:347] Connected to a master server at 127.5.118.190:46607
I20260620 14:51:22.943326 11496 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:22.943523 11496 heartbeater.cc:511] Master 127.5.118.190:46607 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
I20260620 14:51:22.943980 11171 ts_manager.cc:194] Registered new tserver with Master: 22dd6c4747ee48588ea5d9788fb7810e (127.5.118.130:41723)
I20260620 14:51:22.944569 11171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:53809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:51:22.960963 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46607
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:23.043128 11501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:23.043354 11501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:23.043377 11501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:23.044636 11501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:23.044684 11501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:23.044711 11501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:23.046262 11501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:46607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:23.046536 11501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:23.046780 11501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:23.051999 11506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:23.051999 11507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:23.052083 11509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:23.052385 11501 server_base.cc:1034] running on GCE node
I20260620 14:51:23.052616 11501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:23.052892 11501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:23.054069 11501 hybrid_clock.cc:648] HybridClock initialized: now 1781967083054053 us; error 36 us; skew 500 ppm
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:23.056807 11501 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:51:23.057324 11501 webserver.cc:460] Webserver started at http://127.5.118.131:43907/ using document root <none> and password file <none>
I20260620 14:51:23.057593 11501 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:23.057670 11501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:23.057834 11501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:23.058982 11501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "c636d91628674201b071fda480935ca4"
format_stamp: "Formatted at 2026-06-20 14:51:23 on dist-test-slave-2m5f"
server_key: "f7f234b0dc6e62a8f931f09032899764"
server_key_iv: "2c94d4b84b7b59e8a3e34b3ae8672f28"
server_key_version: "encryptionkey@0"
I20260620 14:51:23.059363 11501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "c636d91628674201b071fda480935ca4"
format_stamp: "Formatted at 2026-06-20 14:51:23 on dist-test-slave-2m5f"
server_key: "f7f234b0dc6e62a8f931f09032899764"
server_key_iv: "2c94d4b84b7b59e8a3e34b3ae8672f28"
server_key_version: "encryptionkey@0"
I20260620 14:51:23.060833 11501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 14:51:23.061595 11516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:23.061757 11501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:23.061864 11501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "c636d91628674201b071fda480935ca4"
format_stamp: "Formatted at 2026-06-20 14:51:23 on dist-test-slave-2m5f"
server_key: "f7f234b0dc6e62a8f931f09032899764"
server_key_iv: "2c94d4b84b7b59e8a3e34b3ae8672f28"
server_key_version: "encryptionkey@0"
I20260620 14:51:23.061942 11501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:23.078902 11501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:23.081146 11501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:23.081326 11501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:23.081590 11501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:23.081914 11501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:23.081970 11501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:23.082024 11501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:23.082056 11501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:23.087904 11501 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:45475
I20260620 14:51:23.087958 11629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:45475 every 8 connection(s)
I20260620 14:51:23.088397 11501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:23.096318 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 11501
I20260620 14:51:23.096401 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:23.096609 5594 external_mini_cluster.cc:1428] Setting key ddd81e9af6444882d31bdaba18a3bd4e
I20260620 14:51:23.098644 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:23.089174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:33731 (local address 127.5.118.190:46607)
0620 14:51:23.089219 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:23.089222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:23.089417 (+ 195us) server_negotiation.cc:407] Connection header received
0620 14:51:23.090408 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:23.090409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:23.090430 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:23.090483 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:23.091146 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:23.091749 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:23.093033 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:23.093244 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:23.093333 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:23.095408 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:23.095410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:23.095410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:23.095418 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:23.097187 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:23.097705 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:23.097707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:23.097707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:23.097761 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:23.098023 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:23.098024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:23.098025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:23.098147 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:23.098223 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:23.098479 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 14:51:23.098556 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:51:23.098794 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:23.089157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:46607 (local address 127.5.118.131:33731)
0620 14:51:23.089313 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:23.089348 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:23.090313 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:23.090541 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:23.090545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:23.090853 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:23.091071 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:23.091076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:23.091828 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:23.091829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:23.092911 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:23.092914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:23.093343 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:23.093344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:23.093466 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:23.093931 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:23.093939 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:23.095305 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:23.097261 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:23.097268 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:23.097270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:23.097586 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:23.097827 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:23.097828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:23.097828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:23.097934 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:23.098214 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:23.098218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:23.098339 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 14:51:23.098429 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:51:23.098557 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260620 14:51:23.099282 11630 heartbeater.cc:347] Connected to a master server at 127.5.118.190:46607
I20260620 14:51:23.099392 11630 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:23.099588 11630 heartbeater.cc:511] Master 127.5.118.190:46607 requested a full tablet report, sending...
I20260620 14:51:23.100021 11171 ts_manager.cc:194] Registered new tserver with Master: c636d91628674201b071fda480935ca4 (127.5.118.131:45475)
I20260620 14:51:23.100461 11171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:33731
I20260620 14:51:23.108443 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/test-admin.keytab.
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:23.125300 5594 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:51:23.127060 11637 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:23.126187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.5.118.129:39903)
0620 14:51:23.126392 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:23.126396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:23.126407 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:51:23.126434 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:23.126436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:23.126451 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:23.126523 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:23.126969 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35684: BlockingRecv error: recv got EOF from 127.0.0.1:35684 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":104,"threads_started":1}
W20260620 14:51:23.127259 5594 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:23.128369 5594 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:23.129305 5594 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.129@KRBTEST.COM
I20260620 14:51:23.137022 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:23.129837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.5.118.129:39903)
0620 14:51:23.129923 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:23.129926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:23.129937 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:51:23.130050 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:23.130051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:23.130062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:23.130123 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:23.130526 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:23.131269 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:23.131955 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:23.132339 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:23.132396 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:23.134504 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:23.134511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:23.134511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:23.134517 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:23.135960 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:23.136361 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:23.136363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:23.136364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:23.136421 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:23.136576 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:23.136576 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:23.136577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:23.136705 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:23.136802 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:23.136877 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:51:23.136939 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:51:23.138931 11637 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:23.138407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.5.118.129:39903)
0620 14:51:23.138458 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:23.138460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:23.138493 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:51:23.138540 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:23.138541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:23.138549 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:23.138602 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:23.138860 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35692: BlockingRecv error: recv got EOF from 127.0.0.1:35692 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 14:51:23.139037 5594 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:23.140077 5594 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:23.140985 5594 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.129@KRBTEST.COM
I20260620 14:51:23.148545 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:23.141530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.5.118.129:39903)
0620 14:51:23.141602 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:23.141605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:23.141671 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:51:23.141751 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:23.141752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:23.141761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:23.141818 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:23.142246 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:23.142771 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:23.143479 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:23.143687 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:23.143761 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:23.145957 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:23.145959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:23.145959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:23.145967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:23.147260 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:23.147668 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:23.147669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:23.147669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:23.147719 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:23.147913 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:23.147914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:23.147914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:23.148016 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:23.148121 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:23.148349 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:51:23.148432 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 14:51:23.150439 11637 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:23.149855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.5.118.129:39903)
0620 14:51:23.149906 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:23.149908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:23.149956 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:51:23.150037 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:23.150038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:23.150047 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:23.150098 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:23.150363 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35706: BlockingRecv error: recv got EOF from 127.0.0.1:35706 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 14:51:23.150563 5594 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967053866185-5594-0/krb5kdc/krb5cc)
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:23.151678 5594 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:23.152653 5594 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.129@KRBTEST.COM
I20260620 14:51:23.159677 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:23.153215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.5.118.129:39903)
0620 14:51:23.153268 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:23.153270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:23.153306 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:51:23.153359 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:23.153360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:23.153368 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:23.153420 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:23.153794 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:23.154268 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:23.154942 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:23.155145 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:23.155201 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:23.157360 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:23.157361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:23.157362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:23.157369 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:23.158617 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:23.158958 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:23.158959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:23.158959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:23.159018 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:23.159197 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:23.159198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:23.159198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:23.159295 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:23.159384 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:23.159531 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 14:51:23.159609 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:51:23.160528 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 11233
I20260620 14:51:23.166664 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 11367
I20260620 14:51:23.171543 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 11501
I20260620 14:51:23.177723 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 11139
2026-06-20T14:51:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1897 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11650](info): set up 2 sockets
Jun 20 14:51:23 dist-test-slave-2m5f krb5kdc[11650](info): commencing operation
krb5kdc: starting...
W20260620 14:51:24.225085 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:24 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.190.keytab.
Entry for principal oryx/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.190.keytab.
I20260620 14:51:24.351706 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43647
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.190.keytab
--principal=oryx/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:43647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:24.433719 11663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:24.433996 11663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:24.434036 11663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:24.435235 11663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:24.435318 11663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:24.435343 11663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:24.435371 11663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:24.435420 11663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:24.437106 11663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:43647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.190.keytab
--principal=oryx/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:24.437412 11663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:24.437656 11663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:24.442876 11668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:24.443069 11663 server_base.cc:1034] running on GCE node
W20260620 14:51:24.442852 11671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:24.442878 11669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:24.443444 11663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:24.443783 11663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:24.444948 11663 hybrid_clock.cc:648] HybridClock initialized: now 1781967084444944 us; error 37 us; skew 500 ppm
Jun 20 14:51:24 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:24.447494 11663 init.cc:377] Logged in from keytab as oryx/127.5.118.190@KRBTEST.COM (short username oryx)
I20260620 14:51:24.447953 11663 webserver.cc:460] Webserver started at http://127.5.118.190:44211/ using document root <none> and password file <none>
I20260620 14:51:24.448153 11663 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:24.448258 11663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:24.448361 11663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:24.449391 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "6c664386f7244b1bac4695c11abbe76f"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "3e2c7aaa9f52734e00df723dd4e25ce1"
server_key_iv: "cb3a0c6c3b91209db68760f14f94e6da"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.449712 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "6c664386f7244b1bac4695c11abbe76f"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "3e2c7aaa9f52734e00df723dd4e25ce1"
server_key_iv: "cb3a0c6c3b91209db68760f14f94e6da"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.451031 11663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:24.451807 11678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:24.452032 11663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:24.452128 11663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "6c664386f7244b1bac4695c11abbe76f"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "3e2c7aaa9f52734e00df723dd4e25ce1"
server_key_iv: "cb3a0c6c3b91209db68760f14f94e6da"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.452280 11663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:24.459822 11663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:24.462281 11663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:24.462401 11663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:24.466310 11663 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:43647
I20260620 14:51:24.466358 11730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:43647 every 8 connection(s)
I20260620 14:51:24.466723 11663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:24.467760 11731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:24.470163 11731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f: Bootstrap starting.
I20260620 14:51:24.470781 11731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:24.471076 11731 log.cc:824] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:24.471848 11731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f: No bootstrap required, opened a new log
I20260620 14:51:24.473559 11731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } }
I20260620 14:51:24.473740 11731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:24.473788 11731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c664386f7244b1bac4695c11abbe76f, State: Initialized, Role: FOLLOWER
I20260620 14:51:24.473982 11731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } }
I20260620 14:51:24.474086 11731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:24.474128 11731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:24.474184 11731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:24.475021 11731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } }
I20260620 14:51:24.475152 11731 leader_election.cc:302] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c664386f7244b1bac4695c11abbe76f; no voters:
I20260620 14:51:24.475340 11731 leader_election.cc:288] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:24.475392 11734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:24.475618 11731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:24.475672 11734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 1 LEADER]: Becoming Leader. State: Replica: 6c664386f7244b1bac4695c11abbe76f, State: Running, Role: LEADER
I20260620 14:51:24.475878 11734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } }
I20260620 14:51:24.476317 11736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c664386f7244b1bac4695c11abbe76f. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } }
I20260620 14:51:24.476325 11735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } }
I20260620 14:51:24.476441 11736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:24.476449 11735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:24.476665 11743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:24.476794 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 11663
I20260620 14:51:24.476909 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:24.477119 5594 external_mini_cluster.cc:1428] Setting key 14065080b57859642af55817fec876cb
I20260620 14:51:24.477200 11743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:24.478816 11743 catalog_manager.cc:1347] Generated new cluster ID: ddb2f19ae5394eda94337845625f10d4
I20260620 14:51:24.478873 11743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:51:24 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
I20260620 14:51:24.488997 11743 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:24.489571 11743 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:24.489710 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:24.477767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43518 (local address 127.5.118.190:43647)
0620 14:51:24.478191 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:24.478201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:24.478235 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:51:24.478360 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:24.478367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:24.478422 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:24.478570 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:24.479379 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.480850 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.481893 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.482542 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.482676 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:24.485545 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:24.485550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:24.485551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:24.485563 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:24.487632 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:24.488073 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:24.488076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:24.488076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:24.488223 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:24.488447 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:24.488448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:24.488448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:24.488727 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:24.488893 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:24.489273 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 14:51:24.489427 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":70,"threads_started":1}
I20260620 14:51:24.501744 11743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f: Generated new TSK 0
I20260620 14:51:24.502009 11743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab.
Entry for principal oryx/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab.
I20260620 14:51:24.531540 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43647
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab
--principal=oryx/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:24.615777 11757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:24.616034 11757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:24.616086 11757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:24.617318 11757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:24.617365 11757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:24.617394 11757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:24.618896 11757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab
--principal=oryx/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:24.619153 11757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:24.619403 11757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:24.624878 11765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:24.624878 11763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:24.625054 11757 server_base.cc:1034] running on GCE node
W20260620 14:51:24.625241 11762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:24.625504 11757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:24.625782 11757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:24.626959 11757 hybrid_clock.cc:648] HybridClock initialized: now 1781967084626950 us; error 41 us; skew 500 ppm
Jun 20 14:51:24 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:24.629607 11757 init.cc:377] Logged in from keytab as oryx/127.5.118.129@KRBTEST.COM (short username oryx)
I20260620 14:51:24.630115 11757 webserver.cc:460] Webserver started at http://127.5.118.129:33897/ using document root <none> and password file <none>
I20260620 14:51:24.630345 11757 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:24.630426 11757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:24.630581 11757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:24.631691 11757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "4d9b659fc9ff456c95f4d2228dedf13b"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "1feff5e2a6b5aaa625152865152b9e0d"
server_key_iv: "528ae225387b6ac5988f34a64ee6f6b9"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.632072 11757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "4d9b659fc9ff456c95f4d2228dedf13b"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "1feff5e2a6b5aaa625152865152b9e0d"
server_key_iv: "528ae225387b6ac5988f34a64ee6f6b9"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.633498 11757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:24.634300 11772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:24.634502 11757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:24.634610 11757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "4d9b659fc9ff456c95f4d2228dedf13b"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "1feff5e2a6b5aaa625152865152b9e0d"
server_key_iv: "528ae225387b6ac5988f34a64ee6f6b9"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.634688 11757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:24.642066 11757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:24.644343 11757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:24.644500 11757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:24.644727 11757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:24.645051 11757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:24.645112 11757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:24.645156 11757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:24.645183 11757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:24.651105 11757 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:32795
I20260620 14:51:24.651141 11885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:32795 every 8 connection(s)
I20260620 14:51:24.651489 11757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:24.656747 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 11757
I20260620 14:51:24.656868 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:24.657083 5594 external_mini_cluster.cc:1428] Setting key 35c5dfc88c9f808c0f3f024f3f01b427
Jun 20 14:51:24 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
I20260620 14:51:24.662120 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:24.652221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:54641 (local address 127.5.118.190:43647)
0620 14:51:24.652281 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:24.652283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:24.652492 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:51:24.652991 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:24.652992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:24.653003 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:24.653053 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:24.654026 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.654544 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.655645 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.655851 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.655909 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:24.658777 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:24.658779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:24.658779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:24.658786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:24.660415 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:24.660875 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:24.660876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:24.660876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:24.660946 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:24.661271 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:24.661272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:24.661272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:24.661451 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:24.661540 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:24.661873 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:51:24.661977 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:51:24.662156 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:24.652211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.129:54641)
0620 14:51:24.652365 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:24.652402 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:24.652901 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:24.653127 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:24.653130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:24.653716 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:24.653930 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:24.653935 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.654653 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.654653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:24.655565 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:24.655568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.655928 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.655929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:24.656029 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:24.656690 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:24.656702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:24.658669 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:24.660508 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:24.660517 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:24.660520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:24.660774 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:24.661017 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:24.661018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:24.661018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:24.661144 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:24.661549 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:24.661556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:24.661721 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 14:51:24.661820 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:51:24.661957 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260620 14:51:24.663101 11886 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43647
I20260620 14:51:24.663225 11886 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:24.663424 11886 heartbeater.cc:511] Master 127.5.118.190:43647 requested a full tablet report, sending...
I20260620 14:51:24.664103 11695 ts_manager.cc:194] Registered new tserver with Master: 4d9b659fc9ff456c95f4d2228dedf13b (127.5.118.129:32795)
I20260620 14:51:24.664767 11695 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:54641
WARNING: no policy specified for oryx/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab.
Entry for principal oryx/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab.
I20260620 14:51:24.684294 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43647
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab
--principal=oryx/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:24.772648 11891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:24.772994 11891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:24.773036 11891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:24.774343 11891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:24.774387 11891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:24.774415 11891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:24.776130 11891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab
--principal=oryx/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:24.776432 11891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:24.776677 11891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:51:24.781910 11891 server_base.cc:1034] running on GCE node
W20260620 14:51:24.781883 11897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:24.781893 11899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:24.781876 11896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:24.782338 11891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:24.782608 11891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:24.783758 11891 hybrid_clock.cc:648] HybridClock initialized: now 1781967084783748 us; error 27 us; skew 500 ppm
Jun 20 14:51:24 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:24.786342 11891 init.cc:377] Logged in from keytab as oryx/127.5.118.130@KRBTEST.COM (short username oryx)
I20260620 14:51:24.786839 11891 webserver.cc:460] Webserver started at http://127.5.118.130:39951/ using document root <none> and password file <none>
I20260620 14:51:24.787079 11891 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:24.787148 11891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:24.787286 11891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:24.788460 11891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "12ea2c6b432a49d792f5ebce5457c88d"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "c2bb51e3c0d400b18f3b501bb9de8b2b"
server_key_iv: "2c54fc6a1159899763a19f6e1e4ecfe6"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.788839 11891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "12ea2c6b432a49d792f5ebce5457c88d"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "c2bb51e3c0d400b18f3b501bb9de8b2b"
server_key_iv: "2c54fc6a1159899763a19f6e1e4ecfe6"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.790206 11891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:24.790985 11906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:24.791170 11891 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:24.791270 11891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "12ea2c6b432a49d792f5ebce5457c88d"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "c2bb51e3c0d400b18f3b501bb9de8b2b"
server_key_iv: "2c54fc6a1159899763a19f6e1e4ecfe6"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.791354 11891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:24.809163 11891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:24.811378 11891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:24.811553 11891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:24.811802 11891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:24.812132 11891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:24.812240 11891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:24.812305 11891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:24.812332 11891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:24.818380 11891 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:39953
I20260620 14:51:24.818424 12019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:39953 every 8 connection(s)
I20260620 14:51:24.818760 11891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:24.820003 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 11891
I20260620 14:51:24.820112 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:24.820338 5594 external_mini_cluster.cc:1428] Setting key e8917bc9eafe2a9ba5117a3193f4a101
Jun 20 14:51:24 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.5.118.131@KRBTEST.COM; defaulting to no policy
I20260620 14:51:24.829746 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:24.819552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:42433 (local address 127.5.118.190:43647)
0620 14:51:24.819665 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:24.819668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:24.819874 (+ 206us) server_negotiation.cc:407] Connection header received
0620 14:51:24.820688 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:24.820690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:24.820704 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:24.820751 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:24.822021 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.822520 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.823568 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.823874 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.823949 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:24.826341 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:24.826343 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:24.826343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:24.826353 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:24.828154 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:24.828650 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:24.828652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:24.828652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:24.828723 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:24.828973 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:24.828974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:24.828974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:24.829138 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:24.829218 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:24.829521 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 14:51:24.829631 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:51:24.829993 12022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:24.819514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.130:42433)
0620 14:51:24.819726 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:24.819775 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:24.820560 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:24.821096 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:24.821101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:24.821708 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:24.821911 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:24.821917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.822629 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.822630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:24.823471 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:24.823473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.824002 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.824003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:24.824178 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:24.824853 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:24.824862 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:24.826238 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:24.828270 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:24.828278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:24.828280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:24.828565 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:24.828800 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:24.828801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:24.828801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:24.828901 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:24.829233 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:24.829240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:24.829374 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 14:51:24.829473 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:51:24.829719 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":107,"threads_started":1}
I20260620 14:51:24.830523 12020 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43647
I20260620 14:51:24.830708 12020 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:24.830914 12020 heartbeater.cc:511] Master 127.5.118.190:43647 requested a full tablet report, sending...
I20260620 14:51:24.831305 11695 ts_manager.cc:194] Registered new tserver with Master: 12ea2c6b432a49d792f5ebce5457c88d (127.5.118.130:39953)
I20260620 14:51:24.831749 11695 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:42433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab.
Entry for principal oryx/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab.
I20260620 14:51:24.847426 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43647
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab
--principal=oryx/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:24.930552 12025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:24.930768 12025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:24.930792 12025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:24.932019 12025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:24.932062 12025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:24.932090 12025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:24.933593 12025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab
--principal=oryx/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:24.933794 12025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:24.934001 12025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:24.939332 12031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:24.939360 12025 server_base.cc:1034] running on GCE node
W20260620 14:51:24.939411 12030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:24.939532 12033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:24.939776 12025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:24.939994 12025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:24.941174 12025 hybrid_clock.cc:648] HybridClock initialized: now 1781967084941164 us; error 32 us; skew 500 ppm
Jun 20 14:51:24 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:24.943933 12025 init.cc:377] Logged in from keytab as oryx/127.5.118.131@KRBTEST.COM (short username oryx)
I20260620 14:51:24.944586 12025 webserver.cc:460] Webserver started at http://127.5.118.131:43931/ using document root <none> and password file <none>
I20260620 14:51:24.944837 12025 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:24.944907 12025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:24.945044 12025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:24.946173 12025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "070974bf9d9b4b6ca3168b73c80bad68"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "fd0a700140263a707ba3b7375adfd79a"
server_key_iv: "126a6e889a64d33aa39bbf63d8f281a4"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.946570 12025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "070974bf9d9b4b6ca3168b73c80bad68"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "fd0a700140263a707ba3b7375adfd79a"
server_key_iv: "126a6e889a64d33aa39bbf63d8f281a4"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.947971 12025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:51:24.948906 12040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:24.949115 12025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:24.949223 12025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "070974bf9d9b4b6ca3168b73c80bad68"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "fd0a700140263a707ba3b7375adfd79a"
server_key_iv: "126a6e889a64d33aa39bbf63d8f281a4"
server_key_version: "encryptionkey@0"
I20260620 14:51:24.949314 12025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:24.968077 12025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:24.970351 12025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:24.970593 12025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:24.970904 12025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:24.971268 12025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:24.971331 12025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:24.971376 12025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:24.971403 12025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:24.978940 12025 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:43601
I20260620 14:51:24.978974 12153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:43601 every 8 connection(s)
I20260620 14:51:24.979360 12025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:24.982826 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 12025
I20260620 14:51:24.982940 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:24.983163 5594 external_mini_cluster.cc:1428] Setting key d7205a2b6a0c105a51899d1d70f5fdb0
Jun 20 14:51:24 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
I20260620 14:51:24.989652 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:24.980242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:42765 (local address 127.5.118.190:43647)
0620 14:51:24.980299 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:24.980302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:24.980521 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:51:24.981026 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:24.981028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:24.981040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:24.981115 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:24.982150 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.982682 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.984115 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.984385 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.984438 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:24.986726 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:24.986728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:24.986729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:24.986737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:24.988040 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:24.988589 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:24.988590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:24.988590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:24.988654 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:24.988917 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:24.988918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:24.988919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:24.989051 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:24.989136 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:24.989458 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 14:51:24.989547 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 14:51:24.989850 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:24.980162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.131:42765)
0620 14:51:24.980378 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:24.980418 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:24.980927 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:24.981178 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:24.981181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:24.981847 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:24.982057 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:24.982062 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.982811 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.982812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:24.984008 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:24.984011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:24.984461 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:24.984462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:24.984576 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:24.985183 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:24.985191 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:24.986621 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:24.988131 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:24.988139 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:24.988142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:24.988483 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:24.988726 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:24.988727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:24.988728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:24.988848 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:24.989138 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:24.989145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:24.989316 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 14:51:24.989423 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:51:24.989596 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260620 14:51:24.990376 12154 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43647
I20260620 14:51:24.990513 12154 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:24.990710 12154 heartbeater.cc:511] Master 127.5.118.190:43647 requested a full tablet report, sending...
I20260620 14:51:24.991129 11695 ts_manager.cc:194] Registered new tserver with Master: 070974bf9d9b4b6ca3168b73c80bad68 (127.5.118.131:43601)
I20260620 14:51:24.991477 11695 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:42765
I20260620 14:51:24.995162 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.118.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.189.keytab.
Entry for principal oryx/127.5.118.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.189.keytab.
I20260620 14:51:25.023707 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.189:42711
--webserver_interface=127.5.118.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.189.keytab
--principal=oryx/127.5.118.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:43647,127.5.118.189:42711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:25.106208 12159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:25.106472 12159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:25.106508 12159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:25.107704 12159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:25.107776 12159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:25.107803 12159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:25.107836 12159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:25.107860 12159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:25.109498 12159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:43647,127.5.118.189:42711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.189.keytab
--principal=oryx/127.5.118.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.189:42711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.118.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
W20260620 14:51:25.109665 12159 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.118.190:43647,127.5.118.189:42711'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:51:25.109833 12159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:25.110069 12159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:25.114943 12167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:25.114997 12159 server_base.cc:1034] running on GCE node
W20260620 14:51:25.114936 12165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:25.114933 12164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:25.115432 12159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:25.115723 12159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:25.116891 12159 hybrid_clock.cc:648] HybridClock initialized: now 1781967085116885 us; error 36 us; skew 500 ppm
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:25.119346 12159 init.cc:377] Logged in from keytab as oryx/127.5.118.189@KRBTEST.COM (short username oryx)
I20260620 14:51:25.119870 12159 webserver.cc:460] Webserver started at http://127.5.118.189:40911/ using document root <none> and password file <none>
I20260620 14:51:25.120121 12159 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:25.120210 12159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:25.120373 12159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:25.121538 12159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data/instance:
uuid: "913f34eda30e4e6785bf0c084d487ed8"
format_stamp: "Formatted at 2026-06-20 14:51:25 on dist-test-slave-2m5f"
server_key: "008384f6a6edd0f63d584124bdb743ee"
server_key_iv: "a70fc5fc96e0cbc60fec7e93646fce61"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.121948 12159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/wal/instance:
uuid: "913f34eda30e4e6785bf0c084d487ed8"
format_stamp: "Formatted at 2026-06-20 14:51:25 on dist-test-slave-2m5f"
server_key: "008384f6a6edd0f63d584124bdb743ee"
server_key_iv: "a70fc5fc96e0cbc60fec7e93646fce61"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.123312 12159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:51:25.124080 12174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.124277 12159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:25.124380 12159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/wal
uuid: "913f34eda30e4e6785bf0c084d487ed8"
format_stamp: "Formatted at 2026-06-20 14:51:25 on dist-test-slave-2m5f"
server_key: "008384f6a6edd0f63d584124bdb743ee"
server_key_iv: "a70fc5fc96e0cbc60fec7e93646fce61"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.124466 12159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:25.140691 12159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:25.142953 12159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:25.143134 12159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:25.147516 12159 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.189:42711
I20260620 14:51:25.147579 12226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.189:42711 every 8 connection(s)
I20260620 14:51:25.147966 12159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
I20260620 14:51:25.148962 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 12159
I20260620 14:51:25.149284 12227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:25.151404 12227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.118.189@KRBTEST.COM
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.189@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
I20260620 14:51:25.160285 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.152218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.5.118.190:43647)
0620 14:51:25.152267 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.152269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.152423 (+ 154us) server_negotiation.cc:407] Connection header received
0620 14:51:25.152506 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.152507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.152526 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.152590 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.153074 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.153611 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.154414 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.154685 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.154761 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.157507 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.157508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.157509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.157515 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.158964 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.159357 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.159358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.159359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.159404 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.159610 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.159611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.159611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.159751 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.159804 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.160051 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.160164 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:51:25.160468 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.152122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:43528)
0620 14:51:25.152300 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.152348 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.152440 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.152646 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.152649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.152908 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.152973 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.152975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.153703 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.153704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.154320 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.154322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.154768 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.154769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.154849 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.155371 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.155381 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.157403 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.159053 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.159058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.159059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.159281 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.159466 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.159467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.159467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.159560 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.159819 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.159823 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.159914 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.160012 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.160223 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":23,"thread_start_us":94,"threads_started":1}
I20260620 14:51:25.160468 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.149633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.5.118.189:42711)
0620 14:51:25.149837 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.149848 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.149880 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:51:25.150384 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.150391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.150456 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.150665 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.151248 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.152466 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.153231 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.153726 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.153862 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.156804 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.156811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.156812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.156819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.158451 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.159307 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.159310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.159310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.159413 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.159621 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.159622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.159623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.159912 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.160069 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.160197 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.160261 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260620 14:51:25.161437 12227 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.189@KRBTEST.COM for oryx/127.5.118.189@KRBTEST.COM
I20260620 14:51:25.167595 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.161705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:42711 (local address 127.0.0.1:41554)
0620 14:51:25.161758 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.161769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.161807 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.161919 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.161920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.162038 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.162073 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.162074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.162663 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.162664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.163138 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.163139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.163453 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.163454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.163503 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.163983 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.163988 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.165220 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.166602 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.166602 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.166603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.166764 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.166980 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.166981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.166981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.167035 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.167278 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.167279 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.167314 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.167372 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.167469 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 14:51:25.167595 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.161773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.5.118.189:42711)
0620 14:51:25.161808 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.161810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.161818 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:25.161864 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.161865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.161872 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.161918 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.162133 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.162594 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.163205 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.163394 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.163443 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.165321 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.165323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.165323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.165330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.166492 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.166855 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.166861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.166861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.166912 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.167107 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.167108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.167108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.167213 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.167289 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.167403 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.167515 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":17}
I20260620 14:51:25.169195 12227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: Bootstrap starting.
I20260620 14:51:25.170049 12227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:25.170339 12227 log.cc:824] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:25.171132 12227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: No bootstrap required, opened a new log
I20260620 14:51:25.172670 12227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } }
I20260620 14:51:25.172792 12227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:25.172816 12227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 913f34eda30e4e6785bf0c084d487ed8, State: Initialized, Role: FOLLOWER
I20260620 14:51:25.172976 12227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } }
I20260620 14:51:25.173241 12233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } } }
I20260620 14:51:25.173355 12233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:51:25.173295 12227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:51:25.175235 12244 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:51:25.175307 12244 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:51:25.177199 12159 master_runner.cc:428] Detected that this master 913f34eda30e4e6785bf0c084d487ed8 is joining an existing cluster
I20260620 14:51:25.177258 12159 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.118.189:42711
I20260620 14:51:25.185750 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.178369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.5.118.190:43647)
0620 14:51:25.178442 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.178444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.178587 (+ 143us) server_negotiation.cc:407] Connection header received
0620 14:51:25.178738 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.178740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.178754 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.178816 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.179285 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.180048 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.181218 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.181516 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.181597 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.182519 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.182520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.182521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.182528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.184493 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.184878 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.184879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.184880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.184945 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.185115 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.185116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.185116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.185261 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.185331 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.185575 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.185652 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 14:51:25.185755 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.178313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:43534)
0620 14:51:25.178526 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.178537 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.178645 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.178834 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.178835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.179098 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.179191 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.179193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.180140 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.180142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.181112 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.181114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.181597 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.181598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.181661 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.181985 (+ 324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.181990 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.182429 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.184576 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.184577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.184578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.184814 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.185000 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.185001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.185001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.185054 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.185319 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.185320 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.185356 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.185438 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.185651 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":94,"thread_start_us":159,"threads_started":1}
I20260620 14:51:25.185925 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.178609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:42711 (local address 127.0.0.1:41566)
0620 14:51:25.178762 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.178770 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.178827 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.178953 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.178954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.179167 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.179214 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.179216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.180148 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.180148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.180821 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.180823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.181219 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.181220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.181283 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.181825 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.181830 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.182289 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.184434 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.184435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.184436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.184727 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.184927 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.184928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.184928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.184991 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.185261 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.185262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.185357 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.185433 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.185548 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":53,"thread_start_us":73,"threads_started":1}
I20260620 14:51:25.186738 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.178662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.5.118.189:42711)
0620 14:51:25.178701 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.178703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.178809 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:51:25.178894 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.178895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.178903 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.178949 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.179312 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.180053 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.180922 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.181118 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.181203 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.182391 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.182393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.182393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.182399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.184340 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.184809 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.184810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.184810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.184867 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.185051 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.185052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.185052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.185189 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.185256 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.185456 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.186639 (+ 1183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":795,"server-negotiator.queue_time_us":20}
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.190@KRBTEST.COM for oryx/127.5.118.189@KRBTEST.COM
I20260620 14:51:25.196394 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.188669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:42711 (local address 127.0.0.1:41574)
0620 14:51:25.188874 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.188906 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.188965 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.189103 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.189107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.189350 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.189402 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.189403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.190281 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.190282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.190894 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.190896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.191354 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.191355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.191415 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.191928 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.191933 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.193559 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.195260 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.195267 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.195269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.195473 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.195683 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.195683 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.195684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.195739 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.196015 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.196029 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.196077 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.196157 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.196324 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
I20260620 14:51:25.196398 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.188762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.5.118.189:42711)
0620 14:51:25.188818 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.188819 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.188963 (+ 144us) server_negotiation.cc:407] Connection header received
0620 14:51:25.189022 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.189023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.189033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.189094 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.189470 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.190198 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.190975 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.191279 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.191351 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.193654 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.193656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.193656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.193663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.195177 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.195546 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.195547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.195548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.195616 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.195799 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.195800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.195800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.195932 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.196002 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.196219 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.196299 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 14:51:25.197206 11695 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.118.189:42711: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8"
member_type: NON_VOTER
last_known_addr {
host: "127.5.118.189"
port: 42711
}
attrs {
promote: true
}
}
dest_uuid: "6c664386f7244b1bac4695c11abbe76f"
cas_config_opid_index: -1
I20260620 14:51:25.197530 11695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } }
I20260620 14:51:25.198078 12254 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 913f34eda30e4e6785bf0c084d487ed8 (127.5.118.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } }
I20260620 14:51:25.198405 12201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:25.199229 12254 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.118.189:42711
I20260620 14:51:25.199330 12201 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Refusing update from remote peer 6c664386f7244b1bac4695c11abbe76f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 14:51:25.199393 12254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } }
I20260620 14:51:25.199450 12254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:25.199457 12159 master.cc:537] Master@127.5.118.189:42711 shutting down...
I20260620 14:51:25.199637 12253 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:51:25.200390 11682 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 14:51:25.200803 11682 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f -> Peer 913f34eda30e4e6785bf0c084d487ed8 (127.5.118.189:42711): Couldn't send request to peer 913f34eda30e4e6785bf0c084d487ed8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:51:25.202397 12159 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:25.202489 12159 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:25.202528 12159 tablet_replica.cc:330] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: stopping tablet replica
I20260620 14:51:25.205089 12159 master.cc:552] Master@127.5.118.189:42711 shutdown complete.
I20260620 14:51:25.205139 12159 master_runner.cc:305] Clearing existing system tablet
I20260620 14:51:25.205271 12159 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:25.205319 12159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:25.206055 12159 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:25.206584 12258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.206764 12159 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:25.206825 12159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.206851 12159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/wal
uuid: "913f34eda30e4e6785bf0c084d487ed8"
format_stamp: "Formatted at 2026-06-20 14:51:25 on dist-test-slave-2m5f"
server_key: "008384f6a6edd0f63d584124bdb743ee"
server_key_iv: "a70fc5fc96e0cbc60fec7e93646fce61"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.207620 12159 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:25.209233 12159 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:51:25.209309 12159 log.cc:1196] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:51:25.209517 12159 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: Deleting consensus metadata
I20260620 14:51:25.209645 12159 master_runner.cc:315] Copying system tablet from 127.5.118.190:43647
I20260620 14:51:25.210249 12159 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.118.190:43647
I20260620 14:51:25.216778 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.210725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.5.118.190:43647)
0620 14:51:25.210776 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.210778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.210849 (+ 71us) server_negotiation.cc:407] Connection header received
0620 14:51:25.210912 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.210912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.210921 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.210970 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.211309 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.211945 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.213043 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.213259 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.213318 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.214345 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.214347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.214347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.214353 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.215677 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.216056 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.216057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.216058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.216108 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.216317 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.216318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.216318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.216411 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.216461 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.216657 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.216718 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:51:25.216813 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.210635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:43538)
0620 14:51:25.210768 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.210780 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.210851 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.210976 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.210976 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.211172 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.211229 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.211231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.212023 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.212024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.212956 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.212959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.213336 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.213337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.213425 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.213840 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.213845 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.214239 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.215754 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.215755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.215756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.215977 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.216167 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.216168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.216168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.216255 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.216471 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.216473 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.216521 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.216600 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.216703 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":92,"threads_started":1}
I20260620 14:51:25.217437 11715 tablet_copy_service.cc:140] P 6c664386f7244b1bac4695c11abbe76f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 913f34eda30e4e6785bf0c084d487ed8 ({username='oryx', principal='oryx/127.5.118.189@KRBTEST.COM'} at 127.0.0.1:43538)
I20260620 14:51:25.217542 11715 tablet_copy_service.cc:161] P 6c664386f7244b1bac4695c11abbe76f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 913f34eda30e4e6785bf0c084d487ed8 at {username='oryx', principal='oryx/127.5.118.189@KRBTEST.COM'} at 127.0.0.1:43538: session id = 913f34eda30e4e6785bf0c084d487ed8-00000000000000000000000000000000
I20260620 14:51:25.218210 11715 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:51:25.218905 12159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:51:25.220381 12159 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: tablet copy: Starting download of 0 data blocks...
I20260620 14:51:25.220532 12159 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: tablet copy: Starting download of 1 WAL segments...
I20260620 14:51:25.221547 12159 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:51:25.222414 11715 tablet_copy_service.cc:342] P 6c664386f7244b1bac4695c11abbe76f: Request end of tablet copy session 913f34eda30e4e6785bf0c084d487ed8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.118.189@KRBTEST.COM'} at 127.0.0.1:43538
I20260620 14:51:25.222483 11715 tablet_copy_service.cc:433] P 6c664386f7244b1bac4695c11abbe76f: ending tablet copy session 913f34eda30e4e6785bf0c084d487ed8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 913f34eda30e4e6785bf0c084d487ed8
I20260620 14:51:25.225015 12159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:25.226248 12265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:25.226397 12266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:25.226428 12268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:25.226477 12159 server_base.cc:1034] running on GCE node
I20260620 14:51:25.226624 12159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:25.226758 12159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:25.227888 12159 hybrid_clock.cc:648] HybridClock initialized: now 1781967085227873 us; error 32 us; skew 500 ppm
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:25.229281 12159 init.cc:377] Logged in from keytab as oryx/127.5.118.189@KRBTEST.COM (short username oryx)
I20260620 14:51:25.229563 12159 webserver.cc:460] Webserver started at http://127.5.118.189:44725/ using document root <none> and password file <none>
I20260620 14:51:25.229682 12159 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:25.229719 12159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:25.230422 12159 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:25.230952 12275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.231069 12159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.231137 12159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/wal
uuid: "913f34eda30e4e6785bf0c084d487ed8"
format_stamp: "Formatted at 2026-06-20 14:51:25 on dist-test-slave-2m5f"
server_key: "008384f6a6edd0f63d584124bdb743ee"
server_key_iv: "a70fc5fc96e0cbc60fec7e93646fce61"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.231194 12159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:25.240886 12159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:25.242620 12159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:25.245884 12159 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.189:42711
I20260620 14:51:25.246088 12327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.189:42711 every 8 connection(s)
I20260620 14:51:25.246281 12159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
I20260620 14:51:25.247193 12328 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:51:25.247694 12328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: Bootstrap starting.
I20260620 14:51:25.250649 12328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:51:25.250897 12328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: Bootstrap complete.
I20260620 14:51:25.251081 12328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } }
I20260620 14:51:25.251168 12328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 913f34eda30e4e6785bf0c084d487ed8, State: Initialized, Role: LEARNER
I20260620 14:51:25.251205 12328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } }
I20260620 14:51:25.251385 12328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:25.251425 12331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } }
I20260620 14:51:25.251528 12331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:51:25.252830 12342 catalog_manager.cc:1259] Loaded cluster ID: ddb2f19ae5394eda94337845625f10d4
I20260620 14:51:25.252923 12342 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: loading cluster ID for follower catalog manager: success
I20260620 14:51:25.253824 12342 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: acquiring CA information for follower catalog manager: success
I20260620 14:51:25.254098 12342 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:51:25.269520 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.262152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.5.118.189:42711)
0620 14:51:25.262275 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.262277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.262286 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:25.262310 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.262311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.262320 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.262387 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.263146 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.263787 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.264861 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.265063 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.265131 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.266668 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.266670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.266671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.266677 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.268358 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.268765 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.268766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.268766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.268817 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.269037 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.269037 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.269037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.269140 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.269211 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.269373 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.269435 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.5.118.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.188.keytab.
Entry for principal oryx/127.5.118.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.188.keytab.
I20260620 14:51:25.296007 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.188:45617
--webserver_interface=127.5.118.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.188.keytab
--principal=oryx/127.5.118.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:43647,127.5.118.189:42711,127.5.118.188:45617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:25.382297 12346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:25.382519 12346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:25.382542 12346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:25.383661 12346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:25.383698 12346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:25.383713 12346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:25.383725 12346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:25.383738 12346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:25.385304 12346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:43647,127.5.118.189:42711,127.5.118.188:45617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.188.keytab
--principal=oryx/127.5.118.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.188:45617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.118.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:25.385519 12346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:25.385701 12346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:25.390879 12352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:25.390938 12351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:25.390879 12354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:25.391135 12346 server_base.cc:1034] running on GCE node
I20260620 14:51:25.391460 12346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:25.391883 12346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:25.393051 12346 hybrid_clock.cc:648] HybridClock initialized: now 1781967085393026 us; error 44 us; skew 500 ppm
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:25.395764 12346 init.cc:377] Logged in from keytab as oryx/127.5.118.188@KRBTEST.COM (short username oryx)
I20260620 14:51:25.396361 12346 webserver.cc:460] Webserver started at http://127.5.118.188:37603/ using document root <none> and password file <none>
I20260620 14:51:25.396673 12346 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:25.396744 12346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:25.396893 12346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:25.398017 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data/instance:
uuid: "5b684b23c45942f48630a81de692a1a2"
format_stamp: "Formatted at 2026-06-20 14:51:25 on dist-test-slave-2m5f"
server_key: "34effe4d404124a6544b15c4bf0057f5"
server_key_iv: "9e21343537a0d2ebad8b8bcc5916235d"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.398416 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/wal/instance:
uuid: "5b684b23c45942f48630a81de692a1a2"
format_stamp: "Formatted at 2026-06-20 14:51:25 on dist-test-slave-2m5f"
server_key: "34effe4d404124a6544b15c4bf0057f5"
server_key_iv: "9e21343537a0d2ebad8b8bcc5916235d"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.399838 12346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:25.400836 12361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.401033 12346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:51:25.401140 12346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/wal
uuid: "5b684b23c45942f48630a81de692a1a2"
format_stamp: "Formatted at 2026-06-20 14:51:25 on dist-test-slave-2m5f"
server_key: "34effe4d404124a6544b15c4bf0057f5"
server_key_iv: "9e21343537a0d2ebad8b8bcc5916235d"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.401219 12346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:25.419647 12346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:25.422170 12346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:25.422358 12346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:25.427341 12346 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.188:45617
I20260620 14:51:25.427397 12413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.188:45617 every 8 connection(s)
I20260620 14:51:25.427819 12346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
I20260620 14:51:25.428875 12414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:25.430560 12414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:51:25.431452 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 12346
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.188@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.118.188@KRBTEST.COM
I20260620 14:51:25.440778 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.431505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.5.118.190:43647)
0620 14:51:25.431590 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.431593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.431657 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:51:25.432363 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.432365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.432380 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.432429 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.432992 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.433505 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.434427 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.434627 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.434708 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.437282 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.437284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.437284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.437299 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.439062 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.439513 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.439515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.439515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.439580 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.439916 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.439917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.439917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.440081 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.440150 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.440588 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.440657 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:51:25.441030 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.431334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:43554)
0620 14:51:25.431519 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.431569 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.432247 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.432446 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.432451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.432714 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.432877 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.432881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.433611 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.433612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.434318 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.434321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.434733 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.434734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.434853 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.435437 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.435447 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.437094 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.439166 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.439172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.439174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.439429 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.439654 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.439655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.439655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.439836 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.440166 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.440202 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.440427 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.440562 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.440748 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260620 14:51:25.441115 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.432020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.5.118.188:45617)
0620 14:51:25.432228 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.432233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.432267 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:51:25.432346 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.432348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.432365 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.432464 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.433058 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.433875 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.434620 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.434940 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.435046 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.437734 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.437743 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.437744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.437750 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.439632 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.440144 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.440148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.440149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.440250 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.440456 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.440457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.440457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.440642 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.440797 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.440898 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.440976 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":116,"threads_started":1}
I20260620 14:51:25.441629 12414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.188@KRBTEST.COM for oryx/127.5.118.189@KRBTEST.COM
I20260620 14:51:25.448503 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.441934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:42711 (local address 127.0.0.1:41604)
0620 14:51:25.442015 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.442035 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.442081 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.442238 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.442239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.442390 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.442425 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.442427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.443162 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.443163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.443731 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.443732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.444204 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.444206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.444280 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.444767 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.444771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.446042 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.447478 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.447479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.447479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.447643 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.447845 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.447846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.447847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.447915 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.448252 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.448252 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.448292 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.448350 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.448433 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 14:51:25.448518 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.442041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.5.118.189:42711)
0620 14:51:25.442090 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.442092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.442105 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:51:25.442138 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.442139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.442154 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.442235 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.442526 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.443070 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.443841 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.444078 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.444198 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.446137 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.446139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.446139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.446146 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.447389 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.447714 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.447715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.447715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.447764 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.448013 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.448014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.448015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.448132 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.448261 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.448369 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.448446 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:51:25.449005 12414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.188@KRBTEST.COM for oryx/127.5.118.188@KRBTEST.COM
I20260620 14:51:25.456017 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.449314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:45617 (local address 127.0.0.1:47920)
0620 14:51:25.449349 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.449379 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.449433 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.449630 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.449631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.449773 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.449800 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.449801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.450478 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.450479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.450960 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.450961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.451395 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.451396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.451440 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.451870 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.451874 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.453304 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.454868 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.454868 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.454869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.455031 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.455266 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.455266 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.455267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.455328 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.455715 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.455716 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.455754 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.455863 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.455949 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 14:51:25.456085 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.449348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47920 (local address 127.5.118.188:45617)
0620 14:51:25.449402 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.449404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.449527 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:51:25.449544 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.449545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.449556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.449627 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.449872 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.450348 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.451039 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.451313 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.451396 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.453379 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.453380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.453380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.453386 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.454774 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.455125 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.455126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.455126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.455181 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.455522 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.455522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.455523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.455635 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.455693 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.455898 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.455999 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":37}
I20260620 14:51:25.457551 12414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: Bootstrap starting.
I20260620 14:51:25.458436 12414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:25.459139 12414 log.cc:824] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:25.459975 12414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: No bootstrap required, opened a new log
I20260620 14:51:25.461552 12414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } }
I20260620 14:51:25.461726 12414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:25.461777 12414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b684b23c45942f48630a81de692a1a2, State: Initialized, Role: FOLLOWER
I20260620 14:51:25.461934 12414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } }
I20260620 14:51:25.462212 12419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } } }
I20260620 14:51:25.462288 12414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:25.462306 12419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:51:25.464243 12430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:51:25.464303 12430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:51:25.466950 12346 master_runner.cc:428] Detected that this master 5b684b23c45942f48630a81de692a1a2 is joining an existing cluster
I20260620 14:51:25.467027 12346 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.118.188:45617
I20260620 14:51:25.474715 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.468131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.5.118.188:45617)
0620 14:51:25.468202 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.468204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.468327 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:51:25.468419 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.468420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.468439 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.468470 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.468910 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.469376 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.470142 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.470363 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.470459 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.471585 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.471586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.471587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.471594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.473462 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.473803 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.473804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.473804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.473859 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.474066 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.474067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.474067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.474200 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.474271 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.474540 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.474620 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:51:25.475019 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.468113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:45617 (local address 127.0.0.1:47922)
0620 14:51:25.468268 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.468279 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.468350 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.468531 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.468532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.468766 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.468846 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.468849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.469458 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.469459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.470043 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.470045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.470469 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.470470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.470528 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.471054 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.471059 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.471377 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.473548 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.473549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.473550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.473719 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.473938 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.473939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.473939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.473998 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.474291 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.474292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.474412 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.474834 (+ 422us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.474923 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":39,"thread_start_us":112,"threads_started":1}
I20260620 14:51:25.479496 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.468421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.5.118.190:43647)
0620 14:51:25.468468 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.468470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.472371 (+ 3901us) server_negotiation.cc:407] Connection header received
0620 14:51:25.472504 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.472506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.472529 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.472604 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.473291 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.474063 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.475332 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.475622 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.475710 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.476708 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.476710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.476711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.476718 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.478294 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.478668 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.478669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.478669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.478728 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.478932 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.478933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.478934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.479056 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.479131 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.479315 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.479396 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:51:25.479524 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.468437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:43558)
0620 14:51:25.472280 (+ 3843us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.472292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.472356 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.472610 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.472611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.473125 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.473195 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.473197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.474149 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.474150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.475262 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.475264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.475710 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.475711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.475773 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.476247 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.476251 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.476615 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.478397 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.478398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.478399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.478588 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.478802 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.478803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.478803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.478860 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.479122 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.479123 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.479203 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.479289 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.479397 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3825,"mutex_wait_us":34,"thread_start_us":82,"threads_started":1}
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967084, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.190@KRBTEST.COM for oryx/127.5.118.188@KRBTEST.COM
I20260620 14:51:25.491825 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.481692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:45617 (local address 127.0.0.1:47930)
0620 14:51:25.481756 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.481767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.481809 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.481947 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.481948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.482126 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.482172 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.482174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.482799 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.482800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.483636 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.483638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.483959 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.483960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.484029 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.484466 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.484471 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.485831 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.490604 (+ 4773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.490605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.490606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.490847 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.491053 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.491054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.491055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.491124 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.491412 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.491413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.491467 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.491547 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.491720 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:51:25.491825 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.481752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.5.118.188:45617)
0620 14:51:25.481788 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.481790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.481798 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:25.481880 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.481881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.481889 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.481927 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.482245 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.482719 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.483716 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.483888 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.483941 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.485927 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.485928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.485929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.485936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.490504 (+ 4568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.490923 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.490924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.490925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.490991 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.491193 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.491194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.491195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.491329 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.491401 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.491584 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.491697 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:51:25.492564 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 11757
I20260620 14:51:25.494206 12419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:25.494334 12419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } }
I20260620 14:51:25.494635 12419 leader_election.cc:288] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647), 913f34eda30e4e6785bf0c084d487ed8 (127.5.118.189:42711)
I20260620 14:51:25.495457 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.468678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.5.118.189:42711)
0620 14:51:25.468720 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.468722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.472740 (+ 4018us) server_negotiation.cc:407] Connection header received
0620 14:51:25.472838 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.472839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.472851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.472939 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.488684 (+ 15745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.489208 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.490363 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.490692 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.490771 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.491829 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.491831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.491831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.491839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.494105 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.494551 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.494553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.494554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.494610 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.494826 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.494826 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.494827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.494967 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.495011 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.495225 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.495314 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:51:25.495893 12438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.468758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:42711 (local address 127.0.0.1:41608)
0620 14:51:25.472444 (+ 3686us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.472453 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.472769 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.472944 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.472944 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.488442 (+ 15498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.488523 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.488526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.489296 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.489297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.490264 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.490266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.490781 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.490782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.490862 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.491285 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.491290 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.491732 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.494201 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.494202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.494202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.494468 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.494694 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.494695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.494695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.494756 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.495058 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.495059 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.495113 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.495550 (+ 437us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.495729 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3667,"thread_start_us":70,"threads_started":1}
I20260620 14:51:25.496588 11695 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.118.188:45617: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5b684b23c45942f48630a81de692a1a2"
member_type: NON_VOTER
last_known_addr {
host: "127.5.118.188"
port: 45617
}
attrs {
promote: true
}
}
dest_uuid: "6c664386f7244b1bac4695c11abbe76f"
cas_config_opid_index: 5
I20260620 14:51:25.496824 11695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } }
I20260620 14:51:25.497176 12254 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5b684b23c45942f48630a81de692a1a2 (127.5.118.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:25.498277 12254 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.118.188:45617
I20260620 14:51:25.498368 12254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:25.498464 12254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:25.498843 11705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b684b23c45942f48630a81de692a1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c664386f7244b1bac4695c11abbe76f" is_pre_election: true
I20260620 14:51:25.499215 12302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b684b23c45942f48630a81de692a1a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "913f34eda30e4e6785bf0c084d487ed8" is_pre_election: true
I20260620 14:51:25.499332 12302 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 LEARNER]: Handling vote request from an unknown peer 5b684b23c45942f48630a81de692a1a2
I20260620 14:51:25.499387 12302 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 5b684b23c45942f48630a81de692a1a2 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 14:51:25.499560 12365 leader_election.cc:302] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b684b23c45942f48630a81de692a1a2; no voters: 6c664386f7244b1bac4695c11abbe76f, 913f34eda30e4e6785bf0c084d487ed8
I20260620 14:51:25.499720 12419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:25.500636 12419 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:51:25.500963 12388 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 FOLLOWER]: Refusing update from remote peer 6c664386f7244b1bac4695c11abbe76f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:51:25.501521 12253 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:25.502297 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:32795
--local_ip_for_outbound_sockets=127.5.118.129
--tserver_master_addrs=127.5.118.190:43647
--webserver_port=33897
--webserver_interface=127.5.118.129
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab
--principal=oryx/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.118.190:43647,127.5.118.189:42711
--tserver_master_addrs=127.5.118.190:43647,127.5.118.189:42711,127.5.118.188:45617 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:51:25.502883 12388 raft_consensus.cc:858] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } }
W20260620 14:51:25.502954 12388 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } }
I20260620 14:51:25.503139 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.497253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:42711 (local address 127.0.0.1:41610)
0620 14:51:25.497584 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.497596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.497636 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.499805 (+ 2169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.499806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.499808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:25.499861 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.499863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.500868 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.500869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.502047 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.502050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.502881 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.502882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.502952 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.502976 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.503032 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.503077 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299}
I20260620 14:51:25.503376 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.499618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.5.118.189:42711)
0620 14:51:25.499659 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.499661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.499669 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:25.499692 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.499692 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.499714 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.499786 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:25.499951 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.500775 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.502147 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.502696 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.502766 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.502844 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.503273 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.503319 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:51:25.503566 12302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 LEARNER]: Refusing update from remote peer 6c664386f7244b1bac4695c11abbe76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:51:25.503516 12388 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 913f34eda30e4e6785bf0c084d487ed8 (127.5.118.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } }
I20260620 14:51:25.503789 12253 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 14:51:25.503986 12346 master.cc:537] Master@127.5.118.188:45617 shutting down...
I20260620 14:51:25.504334 12302 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5b684b23c45942f48630a81de692a1a2 (127.5.118.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:25.505646 12331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c664386f7244b1bac4695c11abbe76f. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:25.505752 12331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:51:25.505795 12331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:25.505846 12331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:51:25.505967 12445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } }
I20260620 14:51:25.506062 12445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:51:25.506102 12445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } }
I20260620 14:51:25.506189 12445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:51:25.506676 12419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c664386f7244b1bac4695c11abbe76f. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: NON_VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: true } } }
I20260620 14:51:25.506763 12419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:51:25.507359 12253 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f: attempting to promote NON_VOTER 913f34eda30e4e6785bf0c084d487ed8 to VOTER
I20260620 14:51:25.507586 12253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } }
I20260620 14:51:25.507776 12439 mvcc.cc:205] Tried to move back new op lower bound from 7298937178117144576 to 7298937178013626368. Current Snapshot: MvccSnapshot[applied={T|T < 7298937178022060032}]
I20260620 14:51:25.508145 12302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 LEARNER]: Refusing update from remote peer 6c664386f7244b1bac4695c11abbe76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:51:25.508394 12254 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 14:51:25.508484 11679 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f -> Peer 5b684b23c45942f48630a81de692a1a2 (127.5.118.188:45617): Couldn't send request to peer 5b684b23c45942f48630a81de692a1a2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:51:25.509011 12346 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 14:51:25.509119 12346 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 14:51:25.509155 12346 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: stopping tablet replica
I20260620 14:51:25.509218 12253 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 913f34eda30e4e6785bf0c084d487ed8 (127.5.118.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:25.509590 12302 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 913f34eda30e4e6785bf0c084d487ed8 (127.5.118.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:25.510315 12253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:25.510423 12253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:25.510484 12253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:25.510543 12253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:25.510672 12444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:25.510782 12444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:51:25.524936 12346 master.cc:552] Master@127.5.118.188:45617 shutdown complete.
I20260620 14:51:25.525005 12346 master_runner.cc:305] Clearing existing system tablet
I20260620 14:51:25.525167 12346 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:25.525218 12346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:25.526090 12346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:25.526669 12451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.526780 12346 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:25.526835 12346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:25.526857 12346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/wal
uuid: "5b684b23c45942f48630a81de692a1a2"
format_stamp: "Formatted at 2026-06-20 14:51:25 on dist-test-slave-2m5f"
server_key: "34effe4d404124a6544b15c4bf0057f5"
server_key_iv: "9e21343537a0d2ebad8b8bcc5916235d"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.527644 12346 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:25.529636 12346 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:51:25.529712 12346 log.cc:1196] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:51:25.529937 12346 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: Deleting consensus metadata
I20260620 14:51:25.530047 12346 master_runner.cc:315] Copying system tablet from 127.5.118.190:43647
I20260620 14:51:25.530691 12346 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: tablet copy: Beginning tablet copy session from remote peer at address 127.5.118.190:43647
I20260620 14:51:25.538939 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.531072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:43568)
0620 14:51:25.531234 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.531246 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.531308 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.531483 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.531484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.531691 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.531761 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.531763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.532833 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.532834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.533696 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.533699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.534200 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.534201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.534273 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.535258 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.535264 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.535684 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.537768 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.537770 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.537770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.538011 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.538245 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.538246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.538247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.538309 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.538582 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.538583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.538634 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.538708 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.538824 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":74,"threads_started":1}
I20260620 14:51:25.539070 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.531189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.5.118.190:43647)
0620 14:51:25.531239 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.531241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.531296 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:51:25.531365 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.531366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.531377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.531455 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.531849 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.532731 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.533803 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.534107 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.534190 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.535793 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.535795 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.535796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.535804 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.537663 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.538095 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.538096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.538097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.538173 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.538373 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.538374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.538374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.538515 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.538581 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.538846 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.538977 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:51:25.539866 11715 tablet_copy_service.cc:140] P 6c664386f7244b1bac4695c11abbe76f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5b684b23c45942f48630a81de692a1a2 ({username='oryx', principal='oryx/127.5.118.188@KRBTEST.COM'} at 127.0.0.1:43568)
I20260620 14:51:25.539944 11715 tablet_copy_service.cc:161] P 6c664386f7244b1bac4695c11abbe76f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5b684b23c45942f48630a81de692a1a2 at {username='oryx', principal='oryx/127.5.118.188@KRBTEST.COM'} at 127.0.0.1:43568: session id = 5b684b23c45942f48630a81de692a1a2-00000000000000000000000000000000
I20260620 14:51:25.540589 11715 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:51:25.541086 12346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:51:25.542578 12346 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: tablet copy: Starting download of 0 data blocks...
I20260620 14:51:25.542730 12346 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: tablet copy: Starting download of 1 WAL segments...
I20260620 14:51:25.543597 12346 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:51:25.544548 11715 tablet_copy_service.cc:342] P 6c664386f7244b1bac4695c11abbe76f: Request end of tablet copy session 5b684b23c45942f48630a81de692a1a2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.118.188@KRBTEST.COM'} at 127.0.0.1:43568
I20260620 14:51:25.544608 11715 tablet_copy_service.cc:433] P 6c664386f7244b1bac4695c11abbe76f: ending tablet copy session 5b684b23c45942f48630a81de692a1a2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5b684b23c45942f48630a81de692a1a2
I20260620 14:51:25.547250 12346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:25.548671 12458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:25.549348 12461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:25.549458 12459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:25.549717 12346 server_base.cc:1034] running on GCE node
I20260620 14:51:25.549819 12346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:25.549969 12346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:25.551115 12346 hybrid_clock.cc:648] HybridClock initialized: now 1781967085551106 us; error 26 us; skew 500 ppm
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:25.552743 12346 init.cc:377] Logged in from keytab as oryx/127.5.118.188@KRBTEST.COM (short username oryx)
I20260620 14:51:25.553023 12346 webserver.cc:460] Webserver started at http://127.5.118.188:40475/ using document root <none> and password file <none>
I20260620 14:51:25.553158 12346 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:25.553215 12346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:25.554004 12346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:25.554646 12468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.554822 12346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.554878 12346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/wal
uuid: "5b684b23c45942f48630a81de692a1a2"
format_stamp: "Formatted at 2026-06-20 14:51:25 on dist-test-slave-2m5f"
server_key: "34effe4d404124a6544b15c4bf0057f5"
server_key_iv: "9e21343537a0d2ebad8b8bcc5916235d"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.554957 12346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:25.565586 12346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:25.567489 12346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:25.570660 12346 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.188:45617
I20260620 14:51:25.571044 12346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
I20260620 14:51:25.572009 12521 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:51:25.572590 12521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: Bootstrap starting.
I20260620 14:51:25.573813 12520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.188:45617 every 8 connection(s)
I20260620 14:51:25.575157 12521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:51:25.575393 12521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: Bootstrap complete.
I20260620 14:51:25.575726 12521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } }
I20260620 14:51:25.575824 12521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b684b23c45942f48630a81de692a1a2, State: Initialized, Role: LEARNER
I20260620 14:51:25.575879 12521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } }
I20260620 14:51:25.576061 12521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:25.576077 12524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:25.576315 12524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:51:25.577617 12535 catalog_manager.cc:1259] Loaded cluster ID: ddb2f19ae5394eda94337845625f10d4
I20260620 14:51:25.577701 12535 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: loading cluster ID for follower catalog manager: success
I20260620 14:51:25.579047 12535 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: acquiring CA information for follower catalog manager: success
I20260620 14:51:25.579401 12535 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:51:25.608299 12442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:25.608583 12442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:25.608628 12442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:25.609928 12442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:25.610000 12442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:25.610049 12442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:25.611668 12442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab
--principal=oryx/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:32795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=33897
--tserver_master_addrs=127.5.118.190:43647,127.5.118.189:42711,127.5.118.188:45617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:25.611951 12442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:25.612257 12442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:25.618008 12540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:25.618081 12442 server_base.cc:1034] running on GCE node
W20260620 14:51:25.618029 12541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:25.618261 12543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:25.618510 12442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:25.618777 12442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:25.619982 12442 hybrid_clock.cc:648] HybridClock initialized: now 1781967085619963 us; error 32 us; skew 500 ppm
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:25.622686 12442 init.cc:377] Logged in from keytab as oryx/127.5.118.129@KRBTEST.COM (short username oryx)
I20260620 14:51:25.623165 12442 webserver.cc:460] Webserver started at http://127.5.118.129:33897/ using document root <none> and password file <none>
I20260620 14:51:25.623396 12442 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:25.623454 12442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:25.625088 12442 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:51:25.625738 12550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.625985 12442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:25.626109 12442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "4d9b659fc9ff456c95f4d2228dedf13b"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "1feff5e2a6b5aaa625152865152b9e0d"
server_key_iv: "528ae225387b6ac5988f34a64ee6f6b9"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.626461 12442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:25.637106 12442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:25.639585 12442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:25.639775 12442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:25.640584 12442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:25.640938 12442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:25.640973 12442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.640997 12442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:25.641009 12442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.647444 12442 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:32795
I20260620 14:51:25.647503 12663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:32795 every 8 connection(s)
I20260620 14:51:25.647881 12442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:25.653230 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 12442
I20260620 14:51:25.653400 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 11891
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.188@KRBTEST.COM
I20260620 14:51:25.660087 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.648764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:54187 (local address 127.5.118.190:43647)
0620 14:51:25.648824 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.648827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.649046 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:51:25.650134 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.650135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.650149 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.650195 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.651124 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.651893 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.653020 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.653317 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.653388 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.656801 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.656802 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.656803 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.656811 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.658415 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.658877 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.658878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.658878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.658945 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.659268 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.659269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.659269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.659435 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.659509 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.659822 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.659923 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:51:25.660192 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.648740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:55223 (local address 127.5.118.188:45617)
0620 14:51:25.648895 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.648897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.649046 (+ 149us) server_negotiation.cc:407] Connection header received
0620 14:51:25.650186 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.650187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.650200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.650288 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.651127 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.652292 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.653224 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.654307 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.654398 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.657094 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.657096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.657096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.657105 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.658825 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.659322 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.659323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.659323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.659385 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.659596 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.659597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.659598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.659721 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.659797 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.659999 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.660083 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":52,"threads_started":1}
I20260620 14:51:25.660526 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.648694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:45617 (local address 127.5.118.129:55223)
0620 14:51:25.648961 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.648973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.650023 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.650296 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.650297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.650791 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.651009 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.651015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.652369 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.652370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.653146 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.653148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.654395 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.654396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.654460 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.654911 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.654916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.656981 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.659029 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.659030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.659030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.659260 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.659461 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.659462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.659462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.659520 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.659823 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.659824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.659857 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.659915 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.660342 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":384,"thread_start_us":101,"threads_started":1}
I20260620 14:51:25.660598 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.648694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.129:54187)
0620 14:51:25.648903 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.648948 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.650000 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.650277 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.650281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.650794 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.651009 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.651015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.651986 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.651986 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.652921 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.652923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.653458 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.653459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.653630 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.654305 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.654315 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.656679 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.658523 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.658531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.658534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.658778 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.659021 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.659022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.659022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.659183 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.659496 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.659503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.659685 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.659800 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.659993 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":3,"thread_start_us":104,"threads_started":1}
I20260620 14:51:25.660979 12665 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43647
I20260620 14:51:25.661111 12665 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:25.661306 12665 heartbeater.cc:511] Master 127.5.118.190:43647 requested a full tablet report, sending...
I20260620 14:51:25.661477 12664 heartbeater.cc:347] Connected to a master server at 127.5.118.188:45617
I20260620 14:51:25.661520 12664 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:25.661584 12664 heartbeater.cc:511] Master 127.5.118.188:45617 requested a full tablet report, sending...
I20260620 14:51:25.661728 11695 ts_manager.cc:194] Re-registered known tserver with Master: 4d9b659fc9ff456c95f4d2228dedf13b (127.5.118.129:32795)
I20260620 14:51:25.661993 12485 ts_manager.cc:194] Registered new tserver with Master: 4d9b659fc9ff456c95f4d2228dedf13b (127.5.118.129:32795)
I20260620 14:51:25.662232 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:39953
--local_ip_for_outbound_sockets=127.5.118.130
--tserver_master_addrs=127.5.118.190:43647
--webserver_port=39951
--webserver_interface=127.5.118.130
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab
--principal=oryx/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.118.190:43647,127.5.118.189:42711
--tserver_master_addrs=127.5.118.190:43647,127.5.118.189:42711,127.5.118.188:45617 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:51:25.662241 11695 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:54187
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.189@KRBTEST.COM
I20260620 14:51:25.668723 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.649197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:49349 (local address 127.5.118.189:42711)
0620 14:51:25.649246 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.649248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.649257 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:25.650132 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.650133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.650146 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.650178 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.651118 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.651644 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.661460 (+ 9816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.661818 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.661895 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.665935 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.665937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.665937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.665948 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.667464 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.667858 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.667859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.667860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.667918 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.668123 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.668124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.668124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.668306 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.668386 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.668537 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.668622 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:51:25.668768 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.649031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:42711 (local address 127.5.118.129:49349)
0620 14:51:25.649188 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.649197 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.650023 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.650283 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.650284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.650786 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.651009 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.651014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.651795 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.651795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.652946 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.652948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.663389 (+ 10441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.663390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.663450 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.663995 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.663999 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.665826 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.667570 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.667571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.667571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.667778 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.667995 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.667995 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.667996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.668056 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.668378 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.668379 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.668426 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.668514 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.668643 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260620 14:51:25.669073 12666 heartbeater.cc:347] Connected to a master server at 127.5.118.189:42711
I20260620 14:51:25.669160 12666 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:25.669252 12666 heartbeater.cc:511] Master 127.5.118.189:42711 requested a full tablet report, sending...
I20260620 14:51:25.669638 12292 ts_manager.cc:194] Registered new tserver with Master: 4d9b659fc9ff456c95f4d2228dedf13b (127.5.118.129:32795)
W20260620 14:51:25.747219 12672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:25.747434 12672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:25.747460 12672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:25.748771 12672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:25.748818 12672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:25.748871 12672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:25.750385 12672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab
--principal=oryx/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:39953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=39951
--tserver_master_addrs=127.5.118.190:43647,127.5.118.189:42711,127.5.118.188:45617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:25.750666 12672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:25.750990 12672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:25.756891 12678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:25.756894 12677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:25.757041 12672 server_base.cc:1034] running on GCE node
W20260620 14:51:25.756944 12680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:25.757417 12672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:25.757635 12672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:25.758805 12672 hybrid_clock.cc:648] HybridClock initialized: now 1781967085758799 us; error 36 us; skew 500 ppm
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:25.761653 12672 init.cc:377] Logged in from keytab as oryx/127.5.118.130@KRBTEST.COM (short username oryx)
I20260620 14:51:25.762181 12672 webserver.cc:460] Webserver started at http://127.5.118.130:39951/ using document root <none> and password file <none>
I20260620 14:51:25.762426 12672 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:25.762523 12672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:25.763942 12672 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:25.764623 12687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.764823 12672 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:51:25.764931 12672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "12ea2c6b432a49d792f5ebce5457c88d"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "c2bb51e3c0d400b18f3b501bb9de8b2b"
server_key_iv: "2c54fc6a1159899763a19f6e1e4ecfe6"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.765300 12672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:25.807395 12672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:25.809970 12672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:25.810150 12672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:25.810416 12672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:25.810762 12672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:25.810827 12672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.810870 12672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:25.810896 12672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.817301 12672 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:39953
I20260620 14:51:25.817340 12800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:39953 every 8 connection(s)
I20260620 14:51:25.817714 12672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.188@KRBTEST.COM
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.189@KRBTEST.COM
I20260620 14:51:25.827821 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 12672
I20260620 14:51:25.827994 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 12025
I20260620 14:51:25.829313 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.818513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:58127 (local address 127.5.118.188:45617)
0620 14:51:25.818590 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.818593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.819963 (+ 1370us) server_negotiation.cc:407] Connection header received
0620 14:51:25.820022 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.820023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.820036 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.820136 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.820985 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.821516 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.822605 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.822816 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.822899 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.825512 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.825513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.825514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.825522 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.827424 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.828093 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.828095 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.828095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.828155 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.828471 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.828472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.828472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.828621 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.828667 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.829054 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.829172 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:51:25.829514 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.818634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:53251 (local address 127.5.118.189:42711)
0620 14:51:25.818679 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.818681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.818866 (+ 185us) server_negotiation.cc:407] Connection header received
0620 14:51:25.819860 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.819861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.819872 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.819938 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.820993 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.821752 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.822876 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.823144 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.823211 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.826447 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.826449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.826449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.826457 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.828105 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.828526 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.828527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.828528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.828634 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.828848 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.828849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.828849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.828990 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.829048 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.829327 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.829411 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:51:25.830142 12805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.818471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:42711 (local address 127.5.118.130:53251)
0620 14:51:25.818720 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.818763 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.819731 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.820127 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.820132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.820729 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.820882 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.820889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.821845 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.821846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.822787 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.822790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.823266 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.823267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.823353 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.824005 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.824011 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.826355 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.828224 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.828224 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.828225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.828445 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.828711 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.828712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.828713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.828772 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.829085 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.829086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.829123 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.829208 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.829853 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":106,"thread_start_us":222,"threads_started":1}
I20260620 14:51:25.830178 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.818933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:49503 (local address 127.5.118.190:43647)
0620 14:51:25.818987 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.818989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.820113 (+ 1124us) server_negotiation.cc:407] Connection header received
0620 14:51:25.820251 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.820253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.820267 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.820325 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.820995 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.821761 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.822933 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.823146 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.823227 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.826749 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.826750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.826751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.826758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.828528 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.829307 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.829308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.829308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.829368 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.829584 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.829585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.829585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.829715 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.829761 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.829958 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.830087 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:51:25.830524 12807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.818854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.130:49503)
0620 14:51:25.820062 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.820071 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.820137 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.820351 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.820351 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.820737 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.820882 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.820889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.821847 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.821847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.822867 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.822869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.823224 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.823225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.823282 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.823821 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.823826 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.826680 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.829013 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.829014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.829015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.829223 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.829444 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.829444 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.829445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.829498 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.829809 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.829810 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.829849 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.830290 (+ 441us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.830420 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"mutex_wait_us":55,"thread_start_us":54,"threads_started":1}
I20260620 14:51:25.830156 12806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.818475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:45617 (local address 127.5.118.130:58127)
0620 14:51:25.819883 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.819891 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.819966 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.820149 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.820150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.820662 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.820881 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.820886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.821635 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.821635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.822487 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.822489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.822962 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.822963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.823110 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.823669 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.823678 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.825409 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.827510 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.827518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.827521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.827942 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.828255 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.828256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.828256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.828401 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.828714 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.828720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.828903 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.829842 (+ 939us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.829973 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1386,"thread_start_us":166,"threads_started":1}
I20260620 14:51:25.830989 12802 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43647
I20260620 14:51:25.831104 12802 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:25.831300 12802 heartbeater.cc:511] Master 127.5.118.190:43647 requested a full tablet report, sending...
I20260620 14:51:25.831735 11695 ts_manager.cc:194] Re-registered known tserver with Master: 12ea2c6b432a49d792f5ebce5457c88d (127.5.118.130:39953)
I20260620 14:51:25.832145 11695 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:49503
I20260620 14:51:25.834354 12803 heartbeater.cc:347] Connected to a master server at 127.5.118.189:42711
I20260620 14:51:25.834417 12803 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:25.834470 12803 heartbeater.cc:511] Master 127.5.118.189:42711 requested a full tablet report, sending...
I20260620 14:51:25.834766 12292 ts_manager.cc:194] Registered new tserver with Master: 12ea2c6b432a49d792f5ebce5457c88d (127.5.118.130:39953)
I20260620 14:51:25.835064 12801 heartbeater.cc:347] Connected to a master server at 127.5.118.188:45617
I20260620 14:51:25.835132 12801 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:25.835204 12801 heartbeater.cc:511] Master 127.5.118.188:45617 requested a full tablet report, sending...
I20260620 14:51:25.835371 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:43601
--local_ip_for_outbound_sockets=127.5.118.131
--tserver_master_addrs=127.5.118.190:43647
--webserver_port=43931
--webserver_interface=127.5.118.131
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab
--principal=oryx/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.118.190:43647,127.5.118.189:42711
--tserver_master_addrs=127.5.118.190:43647,127.5.118.189:42711,127.5.118.188:45617 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:51:25.835556 12485 ts_manager.cc:194] Registered new tserver with Master: 12ea2c6b432a49d792f5ebce5457c88d (127.5.118.130:39953)
W20260620 14:51:25.919068 12808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:25.919340 12808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:25.919380 12808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:25.920646 12808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:25.920727 12808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:25.920770 12808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:25.922367 12808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab
--principal=oryx/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:43601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=43931
--tserver_master_addrs=127.5.118.190:43647,127.5.118.189:42711,127.5.118.188:45617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:25.922678 12808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:25.922930 12808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:25.928220 12816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:25.928248 12813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:25.928236 12814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:25.928503 12808 server_base.cc:1034] running on GCE node
I20260620 14:51:25.928754 12808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:25.929055 12808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:25.930226 12808 hybrid_clock.cc:648] HybridClock initialized: now 1781967085930197 us; error 41 us; skew 500 ppm
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:25.932966 12808 init.cc:377] Logged in from keytab as oryx/127.5.118.131@KRBTEST.COM (short username oryx)
I20260620 14:51:25.933419 12808 webserver.cc:460] Webserver started at http://127.5.118.131:43931/ using document root <none> and password file <none>
I20260620 14:51:25.933671 12808 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:25.933763 12808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:25.935174 12808 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:51:25.935885 12823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.936116 12808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:25.936249 12808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "070974bf9d9b4b6ca3168b73c80bad68"
format_stamp: "Formatted at 2026-06-20 14:51:24 on dist-test-slave-2m5f"
server_key: "fd0a700140263a707ba3b7375adfd79a"
server_key_iv: "126a6e889a64d33aa39bbf63d8f281a4"
server_key_version: "encryptionkey@0"
I20260620 14:51:25.936615 12808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:25.959159 12808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:25.961923 12808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:25.962122 12808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:25.962384 12808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:25.962736 12808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:25.962795 12808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.962841 12808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:25.962868 12808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:25.969565 12808 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:43601
I20260620 14:51:25.969610 12936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:43601 every 8 connection(s)
I20260620 14:51:25.970032 12808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:25.970642 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 12808
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.188@KRBTEST.COM
I20260620 14:51:25.982060 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.972101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.5.118.189:42711)
0620 14:51:25.972240 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.972242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.972249 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:51:25.972272 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.972273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.972285 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.973896 (+ 1611us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.973935 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.974715 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.975704 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.975982 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.976072 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.978656 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.978657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.978658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.978665 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.980579 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.980978 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.980979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.980979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.981039 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.981243 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.981244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.981244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.981401 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.981501 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.981849 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.981935 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
Jun 20 14:51:25 dist-test-slave-2m5f krb5kdc[11650](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967085, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.189@KRBTEST.COM
I20260620 14:51:25.984421 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.971668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.5.118.190:43647)
0620 14:51:25.972211 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.972214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.972223 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:25.972255 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.972256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.972269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.972775 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.973202 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.974021 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.975047 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.975394 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.975454 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.977529 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.977530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.977531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.977538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.980772 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.983247 (+ 2475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.983248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.983248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.983313 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.983551 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.983552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.983553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.983692 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.983794 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.984218 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.984295 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260620 14:51:25.986198 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.972247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.5.118.188:45617)
0620 14:51:25.972852 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.972854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.976288 (+ 3434us) server_negotiation.cc:407] Connection header received
0620 14:51:25.976314 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.976315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.976327 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.976383 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.977009 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.977731 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.978799 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.979116 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.979177 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.980774 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.980775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.980775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.980782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.983579 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.984985 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.984985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.984986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.985046 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.985246 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.985247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.985247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.985396 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.985501 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.985968 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.986063 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":59,"threads_started":1}
I20260620 14:51:25.986930 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.971041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:43033 (local address 127.5.118.188:45617)
0620 14:51:25.971087 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.971089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.971312 (+ 223us) server_negotiation.cc:407] Connection header received
0620 14:51:25.972572 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.972573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.972584 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.972616 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.975484 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.976243 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.977461 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.977780 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.977822 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.982777 (+ 4955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.982779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.982779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.982786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.984411 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.984866 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.984867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.984867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.984924 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.985222 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.985223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.985223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.985372 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.985434 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.986776 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.986851 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:51:25.987283 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.970889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:45617 (local address 127.5.118.131:43033)
0620 14:51:25.971176 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.971218 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.972487 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.972695 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.972699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.975275 (+ 2576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.975338 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.975372 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.976362 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.976362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.977333 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.977335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.977926 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.977927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.978082 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.979331 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.979342 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.982668 (+ 3326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.984568 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.984569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.984569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.984773 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.985000 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.985001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.985001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.985138 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.986422 (+ 1284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.986422 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.986617 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.986734 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.986975 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":5,"thread_start_us":86,"threads_started":1}
I20260620 14:51:25.987459 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.970994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:60573 (local address 127.5.118.190:43647)
0620 14:51:25.971053 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.971055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.971384 (+ 329us) server_negotiation.cc:407] Connection header received
0620 14:51:25.972503 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.972504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.972516 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.972568 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.973991 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.974749 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.978616 (+ 3867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.978900 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.978944 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.981659 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.981660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.981660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.981667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.984035 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.985430 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.985430 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.985431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.985484 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.985946 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.985947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.985947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.986235 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.986300 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.987292 (+ 992us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.987381 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:51:25.987609 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.970889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.131:60573)
0620 14:51:25.971329 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.971338 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.972355 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.973076 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.973076 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.973678 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.973870 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.973875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.974876 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.974877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.978461 (+ 3584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.978464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.979018 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.979019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.979084 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.979445 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.979449 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.981541 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.984117 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.984125 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.984128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.984458 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.985553 (+ 1095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.985554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.985554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.985613 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.986335 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.986342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.986617 (+ 275us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.986732 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.987504 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":1053,"thread_start_us":72,"threads_started":1}
I20260620 14:51:25.987903 12937 heartbeater.cc:347] Connected to a master server at 127.5.118.188:45617
I20260620 14:51:25.988210 12937 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:25.988317 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.971463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:49197 (local address 127.5.118.189:42711)
0620 14:51:25.971510 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.971512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.971626 (+ 114us) server_negotiation.cc:407] Connection header received
0620 14:51:25.972505 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.972506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.972514 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.972561 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:25.973991 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.974734 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.978689 (+ 3955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.979934 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.980004 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.984308 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:25.984309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:25.984310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:25.984317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:25.986193 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.987135 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.987137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.987137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.987191 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:25.987534 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:25.987535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:25.987535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:25.987669 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:25.987723 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.988061 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.988168 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:51:25.988423 12937 heartbeater.cc:511] Master 127.5.118.188:45617 requested a full tablet report, sending...
I20260620 14:51:25.988602 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.971303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:42711 (local address 127.5.118.131:49197)
0620 14:51:25.971560 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.971571 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.972365 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.972716 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.972716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.973565 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:25.973870 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.973875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.974872 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.974873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.977332 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.977335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.981412 (+ 4077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.981413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.981486 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.982646 (+ 1160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:25.982651 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:25.984147 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:25.986802 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.986803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.986803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.987048 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.987257 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:25.987258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:25.987258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:25.987314 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:25.987757 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:25.987758 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:25.987798 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.988394 (+ 596us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.988507 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":12,"thread_start_us":59,"threads_started":1}
I20260620 14:51:25.989035 12485 ts_manager.cc:194] Registered new tserver with Master: 070974bf9d9b4b6ca3168b73c80bad68 (127.5.118.131:43601)
I20260620 14:51:25.989065 12939 heartbeater.cc:347] Connected to a master server at 127.5.118.189:42711
I20260620 14:51:25.989115 12939 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:25.989184 12939 heartbeater.cc:511] Master 127.5.118.189:42711 requested a full tablet report, sending...
I20260620 14:51:25.986598 11695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:51:25.989441 12292 ts_manager.cc:194] Registered new tserver with Master: 070974bf9d9b4b6ca3168b73c80bad68 (127.5.118.131:43601)
W20260620 14:51:25.989429 11695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:25.989470 12938 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43647
I20260620 14:51:25.989534 12938 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:25.989600 12938 heartbeater.cc:511] Master 127.5.118.190:43647 requested a full tablet report, sending...
I20260620 14:51:25.989807 11694 ts_manager.cc:194] Re-registered known tserver with Master: 070974bf9d9b4b6ca3168b73c80bad68 (127.5.118.131:43601)
I20260620 14:51:25.990253 11694 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:60573
I20260620 14:51:25.997678 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.993644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:39953 (local address 127.0.0.1:55794)
0620 14:51:25.994014 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.994025 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.994075 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.994425 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.994425 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.994428 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:25.994478 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.994480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.995434 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.995435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.996441 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.996443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.997417 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.997418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.997488 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.997492 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.997540 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.997588 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":208,"thread_start_us":92,"threads_started":1}
I20260620 14:51:25.997689 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.993853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.5.118.130:39953)
0620 14:51:25.994222 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.994227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.994238 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:51:25.994280 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.994282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.994299 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.994402 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:25.994656 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.995337 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.996532 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.997304 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.997420 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.997497 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.997565 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.997603 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":119,"threads_started":1}
I20260620 14:51:25.998230 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.993644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:32795 (local address 127.0.0.1:38954)
0620 14:51:25.993695 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.993706 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.993748 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.994554 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.994555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.994556 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:25.994583 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.994584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.995879 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.995880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.996942 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.996945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.998019 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.998020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.998086 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.998089 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.998119 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.998174 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:51:25.998443 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.994071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:43601 (local address 127.0.0.1:39650)
0620 14:51:25.994201 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:25.994211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:25.994255 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:25.994840 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:25.994841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:25.994843 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:25.994896 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.994898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.996480 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.996480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.997194 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:25.997196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.998204 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.998205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:25.998274 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.998278 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:25.998328 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 14:51:25.998372 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260620 14:51:25.998469 12735 tablet_service.cc:1467] Processing CreateTablet for tablet 63e30b8f8ebf4ca59ea0a9ec48cce194 (DEFAULT_TABLE table=test-table [id=f433c17d74554e9393eaf11bf36c606e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:25.998636 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.993753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.5.118.129:32795)
0620 14:51:25.994398 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.994403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.994412 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:25.994455 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.994457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.994468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.994532 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:25.994667 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.995792 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.997037 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.997924 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.998366 (+ 442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.998453 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.998493 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.998540 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":90,"threads_started":1}
I20260620 14:51:25.998733 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.994154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.5.118.131:43601)
0620 14:51:25.994643 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:25.994647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:25.994656 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:25.994691 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:25.994693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:25.994710 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:25.994784 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:25.994972 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.996393 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.997303 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:25.998023 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:25.998098 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:25.998170 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:25.998614 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 14:51:25.998658 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":79,"threads_started":1}
I20260620 14:51:25.998876 12735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e30b8f8ebf4ca59ea0a9ec48cce194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:25.999337 12871 tablet_service.cc:1467] Processing CreateTablet for tablet 63e30b8f8ebf4ca59ea0a9ec48cce194 (DEFAULT_TABLE table=test-table [id=f433c17d74554e9393eaf11bf36c606e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:25.999657 12871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e30b8f8ebf4ca59ea0a9ec48cce194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:26.001051 12598 tablet_service.cc:1467] Processing CreateTablet for tablet 63e30b8f8ebf4ca59ea0a9ec48cce194 (DEFAULT_TABLE table=test-table [id=f433c17d74554e9393eaf11bf36c606e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:26.001403 12598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e30b8f8ebf4ca59ea0a9ec48cce194. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:26.002960 12963 tablet_bootstrap.cc:492] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: Bootstrap starting.
I20260620 14:51:26.003562 12963 tablet_bootstrap.cc:654] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:26.004295 12963 log.cc:824] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:26.004411 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:26.000287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.5.118.188:45617)
0620 14:51:26.000353 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:26.000355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:26.000832 (+ 477us) server_negotiation.cc:407] Connection header received
0620 14:51:26.000930 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:26.000932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:26.000955 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:26.000994 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:26.001185 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.001884 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.003203 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.003807 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.003858 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:26.003932 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:26.004265 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:51:26.004309 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:51:26.004630 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:25.998779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:45617 (local address 127.0.0.1:47934)
0620 14:51:26.000780 (+ 2001us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:26.000792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:26.000858 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:26.001071 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:26.001072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:26.001073 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:26.001107 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:26.001108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.001986 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.001987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:26.003096 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:26.003099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.004067 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.004068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:26.004137 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:26.004140 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:26.004496 (+ 356us) client_negotiation.cc:240] Negotiation successful
0620 14:51:26.004550 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1990}
I20260620 14:51:26.004642 12964 tablet_bootstrap.cc:492] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: Bootstrap starting.
I20260620 14:51:26.004937 12495 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260620 14:51:26.004961 12966 tablet_bootstrap.cc:492] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: Bootstrap starting.
I20260620 14:51:26.005335 12524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c664386f7244b1bac4695c11abbe76f. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:26.005439 12524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: This master's current role is: LEARNER
I20260620 14:51:26.005602 12966 tablet_bootstrap.cc:654] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:26.005934 12966 log.cc:824] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:26.006129 12964 tablet_bootstrap.cc:654] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:26.006409 12964 log.cc:824] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:26.006773 12966 tablet_bootstrap.cc:492] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: No bootstrap required, opened a new log
I20260620 14:51:26.006847 12966 ts_tablet_manager.cc:1397] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:26.007170 12964 tablet_bootstrap.cc:492] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: No bootstrap required, opened a new log
I20260620 14:51:26.007177 12963 tablet_bootstrap.cc:492] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: No bootstrap required, opened a new log
I20260620 14:51:26.007237 12963 ts_tablet_manager.cc:1397] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:51:26.007258 12964 ts_tablet_manager.cc:1397] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:51:26.009163 12966 raft_consensus.cc:348] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } }
I20260620 14:51:26.009330 12966 raft_consensus.cc:374] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:26.009375 12966 raft_consensus.cc:729] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d9b659fc9ff456c95f4d2228dedf13b, State: Initialized, Role: FOLLOWER
I20260620 14:51:26.009333 12964 raft_consensus.cc:348] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } }
I20260620 14:51:26.009508 12964 raft_consensus.cc:374] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:26.009510 12966 consensus_queue.cc:260] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } }
I20260620 14:51:26.009548 12964 raft_consensus.cc:729] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070974bf9d9b4b6ca3168b73c80bad68, State: Initialized, Role: FOLLOWER
I20260620 14:51:26.009675 12964 consensus_queue.cc:260] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } }
I20260620 14:51:26.009796 12966 ts_tablet_manager.cc:1428] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:26.009728 12963 raft_consensus.cc:348] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } }
I20260620 14:51:26.009877 12963 raft_consensus.cc:374] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:26.009920 12964 ts_tablet_manager.cc:1428] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:26.009920 12963 raft_consensus.cc:729] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ea2c6b432a49d792f5ebce5457c88d, State: Initialized, Role: FOLLOWER
I20260620 14:51:26.010035 12938 heartbeater.cc:503] Master 127.5.118.190:43647 was elected leader, sending a full tablet report...
I20260620 14:51:26.010272 12665 heartbeater.cc:503] Master 127.5.118.190:43647 was elected leader, sending a full tablet report...
I20260620 14:51:26.010751 12963 consensus_queue.cc:260] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } }
I20260620 14:51:26.011386 12963 ts_tablet_manager.cc:1428] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:51:26.011809 12802 heartbeater.cc:503] Master 127.5.118.190:43647 was elected leader, sending a full tablet report...
W20260620 14:51:26.068451 12804 tablet.cc:2369] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:51:26.148860 12667 tablet.cc:2369] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:26.176236 12973 raft_consensus.cc:482] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:26.176425 12973 raft_consensus.cc:504] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } }
I20260620 14:51:26.176918 12973 leader_election.cc:288] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12ea2c6b432a49d792f5ebce5457c88d (127.5.118.130:39953), 070974bf9d9b4b6ca3168b73c80bad68 (127.5.118.131:43601)
I20260620 14:51:26.181010 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:26.177190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:56825 (local address 127.5.118.130:39953)
0620 14:51:26.177250 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:26.177252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:26.177337 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:51:26.177479 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:26.177480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:26.177492 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:26.177551 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:26.177805 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.178403 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.179990 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.180641 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.180708 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:26.180715 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:26.180895 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:51:26.180932 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:51:26.181010 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:26.177543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:38273 (local address 127.5.118.131:43601)
0620 14:51:26.177587 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:26.177590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:26.177598 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:26.177626 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:26.177627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:26.177636 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:26.177702 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:26.177812 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.178372 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.179990 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.180591 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.180679 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:26.180691 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:26.180892 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 14:51:26.180932 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:51:26.181043 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:26.177147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:39953 (local address 127.5.118.129:56825)
0620 14:51:26.177262 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:26.177274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:26.177373 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:26.177616 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:26.177617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:26.177619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:26.177674 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:26.177692 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.178521 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.178522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:26.179876 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:26.179880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.180712 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.180713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:26.180797 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:26.180805 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:51:26.180871 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:51:26.180929 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":74,"thread_start_us":71,"threads_started":1}
I20260620 14:51:26.181046 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:26.177331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:43601 (local address 127.5.118.129:38273)
0620 14:51:26.177445 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:26.177456 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:26.177527 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:26.177717 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:26.177718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:26.177720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:26.177751 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:26.177753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.178469 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.178470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:26.179876 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:26.179880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.180711 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.180712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:26.180797 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:26.180804 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:51:26.180871 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:51:26.180929 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":77,"thread_start_us":64,"threads_started":1}
I20260620 14:51:26.181465 12755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e30b8f8ebf4ca59ea0a9ec48cce194" candidate_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ea2c6b432a49d792f5ebce5457c88d" is_pre_election: true
I20260620 14:51:26.181739 12755 raft_consensus.cc:2454] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d9b659fc9ff456c95f4d2228dedf13b in term 0.
I20260620 14:51:26.181702 12891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e30b8f8ebf4ca59ea0a9ec48cce194" candidate_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070974bf9d9b4b6ca3168b73c80bad68" is_pre_election: true
I20260620 14:51:26.181838 12891 raft_consensus.cc:2454] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d9b659fc9ff456c95f4d2228dedf13b in term 0.
I20260620 14:51:26.182026 12551 leader_election.cc:302] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12ea2c6b432a49d792f5ebce5457c88d, 4d9b659fc9ff456c95f4d2228dedf13b; no voters:
I20260620 14:51:26.182163 12973 raft_consensus.cc:2781] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:26.182255 12973 raft_consensus.cc:482] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:26.182291 12973 raft_consensus.cc:3037] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:26.183024 12973 raft_consensus.cc:504] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } }
I20260620 14:51:26.183195 12973 leader_election.cc:288] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [CANDIDATE]: Term 1 election: Requested vote from peers 12ea2c6b432a49d792f5ebce5457c88d (127.5.118.130:39953), 070974bf9d9b4b6ca3168b73c80bad68 (127.5.118.131:43601)
I20260620 14:51:26.183382 12755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e30b8f8ebf4ca59ea0a9ec48cce194" candidate_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ea2c6b432a49d792f5ebce5457c88d"
I20260620 14:51:26.183452 12891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e30b8f8ebf4ca59ea0a9ec48cce194" candidate_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070974bf9d9b4b6ca3168b73c80bad68"
I20260620 14:51:26.183478 12755 raft_consensus.cc:3037] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:26.183537 12891 raft_consensus.cc:3037] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:26.184423 12891 raft_consensus.cc:2454] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d9b659fc9ff456c95f4d2228dedf13b in term 1.
I20260620 14:51:26.184423 12755 raft_consensus.cc:2454] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d9b659fc9ff456c95f4d2228dedf13b in term 1.
I20260620 14:51:26.184587 12551 leader_election.cc:302] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 070974bf9d9b4b6ca3168b73c80bad68, 4d9b659fc9ff456c95f4d2228dedf13b; no voters:
I20260620 14:51:26.184674 12973 raft_consensus.cc:2781] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:26.184800 12973 raft_consensus.cc:686] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 1 LEADER]: Becoming Leader. State: Replica: 4d9b659fc9ff456c95f4d2228dedf13b, State: Running, Role: LEADER
I20260620 14:51:26.184903 12973 consensus_queue.cc:237] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } }
I20260620 14:51:26.185853 11693 catalog_manager.cc:5495] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d9b659fc9ff456c95f4d2228dedf13b (127.5.118.129). New cstate: current_term: 1 leader_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } health_report { overall_health: HEALTHY } } }
I20260620 14:51:26.186506 12449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: NON_VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: true } } }
I20260620 14:51:26.186628 12449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:26.187512 12980 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f: attempting to promote NON_VOTER 5b684b23c45942f48630a81de692a1a2 to VOTER
I20260620 14:51:26.187637 12980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } }
I20260620 14:51:26.188237 12495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 LEARNER]: Refusing update from remote peer 6c664386f7244b1bac4695c11abbe76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 14:51:26.188272 12302 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Refusing update from remote peer 6c664386f7244b1bac4695c11abbe76f: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 14:51:26.188464 12254 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 14:51:26.188560 12449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 14:51:26.189204 12980 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 5b684b23c45942f48630a81de692a1a2 (127.5.118.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:26.189321 12302 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 5b684b23c45942f48630a81de692a1a2 (127.5.118.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:26.189379 12495 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 5b684b23c45942f48630a81de692a1a2 (127.5.118.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:26.190197 12980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:26.190238 12254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:26.190285 12980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:26.190308 12254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:26.190367 12254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:26.190456 12254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:26.190557 12982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:26.190649 12982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:51:26.190863 12524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c664386f7244b1bac4695c11abbe76f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:26.190971 12524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:51:26.221524 12940 tablet.cc:2369] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:26.227717 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:26.225309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.5.118.129:32795)
0620 14:51:26.225374 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:26.225376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:26.225433 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:51:26.225493 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:26.225494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:26.225511 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:26.225627 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:26.225748 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.226234 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.226935 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.227184 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.227251 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:26.227405 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:26.227512 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:26.227585 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:26.227634 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:51:26.227654 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:51:26.230223 12891 raft_consensus.cc:1261] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68 [term 1 FOLLOWER]: Refusing update from remote peer 4d9b659fc9ff456c95f4d2228dedf13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:26.230291 12755 raft_consensus.cc:1261] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d [term 1 FOLLOWER]: Refusing update from remote peer 4d9b659fc9ff456c95f4d2228dedf13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:26.230680 12973 consensus_queue.cc:1035] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:26.230849 12978 consensus_queue.cc:1035] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:26.233604 12984 mvcc.cc:205] Tried to move back new op lower bound from 7298937185196318720 to 7298937185013981184. Current Snapshot: MvccSnapshot[applied={T|T < 7298937185196318720}]
I20260620 14:51:26.235002 12985 mvcc.cc:205] Tried to move back new op lower bound from 7298937185196318720 to 7298937185013981184. Current Snapshot: MvccSnapshot[applied={T|T < 7298937185196318720}]
I20260620 14:51:26.237241 11694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:26.237345 11694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:26.238945 11694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c664386f7244b1bac4695c11abbe76f: Sending DeleteTablet for 3 replicas of tablet 63e30b8f8ebf4ca59ea0a9ec48cce194
I20260620 14:51:26.239338 12598 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e30b8f8ebf4ca59ea0a9ec48cce194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:26 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:38954
I20260620 14:51:26.239573 12993 tablet_replica.cc:330] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: stopping tablet replica
I20260620 14:51:26.239586 12735 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e30b8f8ebf4ca59ea0a9ec48cce194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:26 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:55794
I20260620 14:51:26.239799 12993 raft_consensus.cc:2229] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:26.239887 12871 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e30b8f8ebf4ca59ea0a9ec48cce194 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:26 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:39650
I20260620 14:51:26.239952 12993 raft_consensus.cc:2258] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:26.240142 12994 tablet_replica.cc:330] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: stopping tablet replica
I20260620 14:51:26.240293 12994 raft_consensus.cc:2229] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:26.240418 12994 raft_consensus.cc:2258] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:26.240759 12993 ts_tablet_manager.cc:1905] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:26.240831 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 11663
I20260620 14:51:26.243257 12993 ts_tablet_manager.cc:1918] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:26.243412 12993 log.cc:1196] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-0/wal/wals/63e30b8f8ebf4ca59ea0a9ec48cce194
I20260620 14:51:26.243748 12993 ts_tablet_manager.cc:1939] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 4d9b659fc9ff456c95f4d2228dedf13b: Deleting consensus metadata
I20260620 14:51:26.244719 12996 tablet_replica.cc:330] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: stopping tablet replica
I20260620 14:51:26.244844 12996 raft_consensus.cc:2229] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:26.244961 12996 raft_consensus.cc:2258] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:26.245473 12996 ts_tablet_manager.cc:1905] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:26.247272 12996 ts_tablet_manager.cc:1918] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:26.247349 12996 log.cc:1196] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/63e30b8f8ebf4ca59ea0a9ec48cce194
I20260620 14:51:26.247666 12996 ts_tablet_manager.cc:1939] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 070974bf9d9b4b6ca3168b73c80bad68: Deleting consensus metadata
W20260620 14:51:26.248462 12827 connection.cc:535] server connection from 127.0.0.1:39650 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 14:51:26.248771 12554 connection.cc:535] server connection from 127.0.0.1:38954 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:51:26.249800 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:26.240486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.5.118.188:45617)
0620 14:51:26.240532 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:26.240535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:26.241111 (+ 576us) server_negotiation.cc:407] Connection header received
0620 14:51:26.241272 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:26.241273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:26.241284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:26.241314 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:26.242122 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.242858 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.243983 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.244291 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.244327 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:26.246284 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:26.246285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:26.246286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:26.246294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:26.248348 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:26.248842 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:26.248844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:26.248844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:26.248908 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:26.249121 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:26.249122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:26.249122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:26.249268 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:26.249364 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:26.249584 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 14:51:26.249672 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:51:26.251359 12994 ts_tablet_manager.cc:1905] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:26.253228 12994 ts_tablet_manager.cc:1918] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:26.253315 12994 log.cc:1196] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/63e30b8f8ebf4ca59ea0a9ec48cce194
I20260620 14:51:26.253599 12994 ts_tablet_manager.cc:1939] T 63e30b8f8ebf4ca59ea0a9ec48cce194 P 12ea2c6b432a49d792f5ebce5457c88d: Deleting consensus metadata
W20260620 14:51:26.254051 12691 connection.cc:441] server connection from 127.0.0.1:55794 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55794 (request call id 1) could send its response
I20260620 14:51:26.259733 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:26.250705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.5.118.188:45617)
0620 14:51:26.250876 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:26.250878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:26.250922 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:51:26.251044 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:26.251046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:26.251068 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:26.251099 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:26.252344 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.253035 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.254075 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.254358 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.254401 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:26.256601 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:26.256602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:26.256602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:26.256609 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:26.258329 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:26.258881 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:26.258882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:26.258882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:26.258947 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:26.259168 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:26.259169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:26.259169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:26.259290 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:26.259396 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:26.259539 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:51:26.259629 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260620 14:51:26.260329 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:26.250884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.5.118.189:42711)
0620 14:51:26.250925 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:26.250927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:26.251088 (+ 161us) server_negotiation.cc:407] Connection header received
0620 14:51:26.251183 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:26.251184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:26.251195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:26.251241 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:26.252011 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.252872 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.254067 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:26.254358 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:26.254401 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:26.256492 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:26.256494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:26.256494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:26.256503 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:26.258329 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:26.258863 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:26.258864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:26.258864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:26.258929 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:26.259147 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:26.259148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:26.259148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:26.259302 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:26.259398 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:26.260100 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 14:51:26.260233 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:51:27.235335 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.235011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.130:47095)
0620 14:51:27.235193 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.235271 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260620 14:51:27.235504 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.235195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.129:55159)
0620 14:51:27.235319 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.235463 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
W20260620 14:51:27.235666 12802 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:43647 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
W20260620 14:51:27.235737 12665 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:43647 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
I20260620 14:51:27.236331 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.236048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.131:42481)
0620 14:51:27.236211 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.236288 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":55,"threads_started":1}
W20260620 14:51:27.236518 12938 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:43647 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
I20260620 14:51:27.740031 13006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c664386f7244b1bac4695c11abbe76f)
I20260620 14:51:27.740054 13007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c664386f7244b1bac4695c11abbe76f)
I20260620 14:51:27.740156 13006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } }
I20260620 14:51:27.740226 13007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } }
I20260620 14:51:27.740419 13006 leader_election.cc:288] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647), 913f34eda30e4e6785bf0c084d487ed8 (127.5.118.189:42711)
I20260620 14:51:27.740513 13007 leader_election.cc:288] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647), 5b684b23c45942f48630a81de692a1a2 (127.5.118.188:45617)
I20260620 14:51:27.740837 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.740586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:44080)
0620 14:51:27.740724 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.740790 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":47,"threads_started":1}
I20260620 14:51:27.740859 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.740606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:44088)
0620 14:51:27.740790 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.740826 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":47,"threads_started":1}
W20260620 14:51:27.740986 12472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
W20260620 14:51:27.741151 12279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
I20260620 14:51:27.741647 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.741477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:44094)
0620 14:51:27.741594 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.741619 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
W20260620 14:51:27.741786 12279 leader_election.cc:334] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647): Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
I20260620 14:51:27.741886 13011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.741768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:44108)
0620 14:51:27.741825 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.741857 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 14:51:27.741945 12472 leader_election.cc:334] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647): Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
I20260620 14:51:27.743867 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.740730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:42711 (local address 127.0.0.1:41624)
0620 14:51:27.740911 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.740921 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:27.740982 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:27.741219 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:27.741220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:27.741223 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:27.741282 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:27.741283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.742206 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.742207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:27.743047 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:27.743050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.743620 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.743621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:27.743684 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:27.743688 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:27.743754 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:51:27.743796 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":51,"threads_started":1}
I20260620 14:51:27.743939 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.740644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:45617 (local address 127.0.0.1:47958)
0620 14:51:27.741035 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.741045 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:27.741166 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:27.741376 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:27.741377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:27.741379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:27.741428 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:27.741430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.742281 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.742283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:27.743075 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:27.743077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.743667 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.743668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:27.743735 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:27.743738 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:27.743804 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:51:27.743852 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":40,"threads_started":1}
I20260620 14:51:27.744210 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.740827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.5.118.188:45617)
0620 14:51:27.740911 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:27.740913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:27.741136 (+ 223us) server_negotiation.cc:407] Connection header received
0620 14:51:27.741252 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:27.741253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:27.741263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:27.741330 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:27.741494 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.742193 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.743186 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.743566 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.743661 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:27.743672 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:27.744089 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 14:51:27.744137 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61,"thread_start_us":33,"threads_started":1}
I20260620 14:51:27.744307 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.740848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.5.118.189:42711)
0620 14:51:27.740974 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:27.740976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:27.740984 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:27.741084 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:27.741085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:27.741095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:27.741161 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:27.741362 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.742121 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.743147 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.743522 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.743617 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:27.743628 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:27.743768 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:51:27.743803 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":40,"threads_started":1}
I20260620 14:51:27.744401 12495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "913f34eda30e4e6785bf0c084d487ed8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5b684b23c45942f48630a81de692a1a2" is_pre_election: true
I20260620 14:51:27.744457 12302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b684b23c45942f48630a81de692a1a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "913f34eda30e4e6785bf0c084d487ed8" is_pre_election: true
I20260620 14:51:27.744505 12495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913f34eda30e4e6785bf0c084d487ed8 in term 1.
I20260620 14:51:27.744532 12302 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b684b23c45942f48630a81de692a1a2 in term 1.
I20260620 14:51:27.744666 12472 leader_election.cc:302] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b684b23c45942f48630a81de692a1a2, 913f34eda30e4e6785bf0c084d487ed8; no voters: 6c664386f7244b1bac4695c11abbe76f
I20260620 14:51:27.744726 12276 leader_election.cc:302] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b684b23c45942f48630a81de692a1a2, 913f34eda30e4e6785bf0c084d487ed8; no voters: 6c664386f7244b1bac4695c11abbe76f
I20260620 14:51:27.744750 13006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:51:27.744791 13006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c664386f7244b1bac4695c11abbe76f)
I20260620 14:51:27.744820 13006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:51:27.744831 13007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:51:27.744869 13007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c664386f7244b1bac4695c11abbe76f)
I20260620 14:51:27.744899 13007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:51:27.745596 13007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } }
I20260620 14:51:27.745631 13006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } }
I20260620 14:51:27.745774 13006 leader_election.cc:288] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [CANDIDATE]: Term 2 election: Requested vote from peers 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647), 913f34eda30e4e6785bf0c084d487ed8 (127.5.118.189:42711)
I20260620 14:51:27.745769 13007 leader_election.cc:288] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 2 election: Requested vote from peers 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647), 5b684b23c45942f48630a81de692a1a2 (127.5.118.188:45617)
I20260620 14:51:27.745942 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.745841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:44126)
0620 14:51:27.745881 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.745914 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 14:51:27.745966 12495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "913f34eda30e4e6785bf0c084d487ed8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5b684b23c45942f48630a81de692a1a2"
I20260620 14:51:27.746060 12495 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 913f34eda30e4e6785bf0c084d487ed8 in current term 2: Already voted for candidate 5b684b23c45942f48630a81de692a1a2 in this term.
I20260620 14:51:27.746129 12302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b684b23c45942f48630a81de692a1a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "913f34eda30e4e6785bf0c084d487ed8"
I20260620 14:51:27.746212 12302 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b684b23c45942f48630a81de692a1a2 in current term 2: Already voted for candidate 913f34eda30e4e6785bf0c084d487ed8 in this term.
I20260620 14:51:27.746191 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.745956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:44128)
0620 14:51:27.746135 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.746164 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:51:27.746417 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.746337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:44132)
0620 14:51:27.746370 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.746396 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 14:51:27.746493 12279 leader_election.cc:334] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647): Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
I20260620 14:51:27.746558 12279 leader_election.cc:302] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 913f34eda30e4e6785bf0c084d487ed8; no voters: 5b684b23c45942f48630a81de692a1a2, 6c664386f7244b1bac4695c11abbe76f
I20260620 14:51:27.746585 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.746500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:44144)
0620 14:51:27.746533 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.746561 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 14:51:27.746656 12472 leader_election.cc:334] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647): Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
I20260620 14:51:27.746666 13007 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 14:51:27.746735 12472 leader_election.cc:302] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b684b23c45942f48630a81de692a1a2; no voters: 6c664386f7244b1bac4695c11abbe76f, 913f34eda30e4e6785bf0c084d487ed8
I20260620 14:51:27.746819 13006 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 14:51:28.236356 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.236028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.130:35675)
0620 14:51:28.236283 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:28.236315 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":172,"threads_started":1}
I20260620 14:51:28.236330 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.236029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.129:57543)
0620 14:51:28.236214 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:28.236270 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":113,"threads_started":1}
I20260620 14:51:28.237056 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.236853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.131:48897)
0620 14:51:28.236973 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:28.237006 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260620 14:51:29.237011 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.236759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.129:48363)
0620 14:51:29.236904 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.236951 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260620 14:51:29.237044 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.236846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.130:52475)
0620 14:51:29.236954 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.236998 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260620 14:51:29.237620 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.237409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.131:57289)
0620 14:51:29.237515 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.237555 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":48,"threads_started":1}
I20260620 14:51:29.429150 13023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:29.429250 13023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } }
I20260620 14:51:29.429593 13023 leader_election.cc:288] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647), 5b684b23c45942f48630a81de692a1a2 (127.5.118.188:45617)
I20260620 14:51:29.429812 12495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "913f34eda30e4e6785bf0c084d487ed8" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5b684b23c45942f48630a81de692a1a2" is_pre_election: true
I20260620 14:51:29.429855 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.429672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:40494)
0620 14:51:29.429773 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.429818 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":65,"threads_started":1}
I20260620 14:51:29.429896 12495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 913f34eda30e4e6785bf0c084d487ed8 in term 2.
I20260620 14:51:29.430127 12276 leader_election.cc:302] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b684b23c45942f48630a81de692a1a2, 913f34eda30e4e6785bf0c084d487ed8; no voters:
I20260620 14:51:29.430230 13023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 14:51:29.430274 13023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:29.430289 13023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:51:29.430368 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.430211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:40502)
0620 14:51:29.430254 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.430278 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 14:51:29.430481 12279 leader_election.cc:334] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647): Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
I20260620 14:51:29.431147 13023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } }
I20260620 14:51:29.431284 13023 leader_election.cc:288] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 3 election: Requested vote from peers 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647), 5b684b23c45942f48630a81de692a1a2 (127.5.118.188:45617)
I20260620 14:51:29.431437 12495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "913f34eda30e4e6785bf0c084d487ed8" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5b684b23c45942f48630a81de692a1a2"
I20260620 14:51:29.431470 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.431373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:40512)
0620 14:51:29.431411 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.431441 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 14:51:29.431532 12495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:51:29.431802 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.431692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:40520)
0620 14:51:29.431746 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.431771 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:51:29.431907 12279 leader_election.cc:334] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647): Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
I20260620 14:51:29.432300 12495 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 913f34eda30e4e6785bf0c084d487ed8 in term 3.
I20260620 14:51:29.432489 12276 leader_election.cc:302] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b684b23c45942f48630a81de692a1a2, 913f34eda30e4e6785bf0c084d487ed8; no voters: 6c664386f7244b1bac4695c11abbe76f
I20260620 14:51:29.432574 13023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:51:29.432672 13023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [term 3 LEADER]: Becoming Leader. State: Replica: 913f34eda30e4e6785bf0c084d487ed8, State: Running, Role: LEADER
I20260620 14:51:29.432749 13023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } }
I20260620 14:51:29.433115 13026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 913f34eda30e4e6785bf0c084d487ed8. Latest consensus state: current_term: 3 leader_uuid: "913f34eda30e4e6785bf0c084d487ed8" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:29.433228 13026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:29.433408 13028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:29.433794 13028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:29.433848 13028 catalog_manager.cc:1259] Loaded cluster ID: ddb2f19ae5394eda94337845625f10d4
I20260620 14:51:29.433866 13028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:29.434001 13028 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:29.434053 13028 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: Loaded TSK: 0
I20260620 14:51:29.434142 13028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:51:29.506480 12291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:29.508109 12291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:29.509220 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.509100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:40542)
0620 14:51:29.509158 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.509195 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:51:29.509232 12495 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [term 3 FOLLOWER]: Refusing update from remote peer 913f34eda30e4e6785bf0c084d487ed8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 14:51:29.509469 13026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 14:51:29.509644 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.509534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.0.0.1:40554)
0620 14:51:29.509588 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.509617 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:51:29.509719 12279 consensus_peers.cc:487] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 -> Peer 6c664386f7244b1bac4695c11abbe76f (127.5.118.190:43647): Couldn't send request to peer 6c664386f7244b1bac4695c11abbe76f. Status: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:51:29.510110 13030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 913f34eda30e4e6785bf0c084d487ed8. Latest consensus state: current_term: 3 leader_uuid: "913f34eda30e4e6785bf0c084d487ed8" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:29.510205 13030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:51:29.510676 13029 mvcc.cc:205] Tried to move back new op lower bound from 7298937198627549184 to 7298937198317137920. Current Snapshot: MvccSnapshot[applied={T|T < 7298937198627549184}]
I20260620 14:51:29.511065 13026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "913f34eda30e4e6785bf0c084d487ed8" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:29.511075 13030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "913f34eda30e4e6785bf0c084d487ed8" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "6c664386f7244b1bac4695c11abbe76f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43647 } } peers { permanent_uuid: "913f34eda30e4e6785bf0c084d487ed8" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 42711 } attrs { promote: false } } peers { permanent_uuid: "5b684b23c45942f48630a81de692a1a2" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 45617 } attrs { promote: false } } }
I20260620 14:51:29.511157 13026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:29.511173 13030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b684b23c45942f48630a81de692a1a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:51:29.516633 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.513120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:32795 (local address 127.0.0.1:47580)
0620 14:51:29.513156 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.513168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:29.513221 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:29.513749 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:29.513750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:29.513753 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:29.513808 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.513810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.514496 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.514497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.515655 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.515657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.516387 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.516388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.516453 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.516458 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:51:29.516507 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:51:29.516552 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 14:51:29.516726 13035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.513157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:43601 (local address 127.0.0.1:36166)
0620 14:51:29.513333 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.513343 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:29.513485 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:29.513783 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:29.513783 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:29.513785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:29.513812 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.513813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.514794 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.514796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.515699 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.515701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.516477 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.516478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.516532 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.516535 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:29.516578 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 14:51:29.516610 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260620 14:51:29.516911 13038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.513367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:39953 (local address 127.0.0.1:36308)
0620 14:51:29.513468 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.513477 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:29.513530 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:29.513819 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:29.513820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:29.513822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:29.513871 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.513872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.514766 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.514767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.515858 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.515860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.516656 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.516657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.516720 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.516725 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:51:29.516777 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:51:29.516824 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":28,"thread_start_us":62,"threads_started":1}
I20260620 14:51:29.516952 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.513312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.5.118.129:32795)
0620 14:51:29.513629 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.513632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.513641 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:29.513668 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.513669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.513682 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.513749 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:29.513867 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.514407 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.515751 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.516296 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.516395 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.516410 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.516810 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.516859 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":63,"threads_started":1}
I20260620 14:51:29.517125 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.513526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36308 (local address 127.5.118.130:39953)
0620 14:51:29.513665 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.513667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.513675 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:29.513701 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.513702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.513713 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.513779 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:29.513933 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.514683 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.515947 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.516552 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.516614 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.516624 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.516990 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.517046 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":56,"threads_started":1}
I20260620 14:51:29.517134 12598 tablet_service.cc:1467] Processing CreateTablet for tablet 93768df37e454690b2faeec227bb8d7d (DEFAULT_TABLE table=test-table [id=16a852bf6dd442899058461c54dea1ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:29.517278 12598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93768df37e454690b2faeec227bb8d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:29.517452 12735 tablet_service.cc:1467] Processing CreateTablet for tablet 93768df37e454690b2faeec227bb8d7d (DEFAULT_TABLE table=test-table [id=16a852bf6dd442899058461c54dea1ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:29.517580 12735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93768df37e454690b2faeec227bb8d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:29.517971 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.513220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.5.118.131:43601)
0620 14:51:29.513642 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.513644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.513652 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:29.513679 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.513680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.513690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.513873 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:29.513908 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.514684 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.515788 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.516398 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.516479 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.516491 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.516600 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.516640 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":92,"threads_started":1}
I20260620 14:51:29.518365 12871 tablet_service.cc:1467] Processing CreateTablet for tablet 93768df37e454690b2faeec227bb8d7d (DEFAULT_TABLE table=test-table [id=16a852bf6dd442899058461c54dea1ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:29.518409 13040 tablet_bootstrap.cc:492] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b: Bootstrap starting.
I20260620 14:51:29.518476 12871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93768df37e454690b2faeec227bb8d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:29.518738 13040 tablet_bootstrap.cc:654] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:29.519124 13042 tablet_bootstrap.cc:492] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d: Bootstrap starting.
I20260620 14:51:29.519308 13040 tablet_bootstrap.cc:492] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b: No bootstrap required, opened a new log
I20260620 14:51:29.519354 13040 ts_tablet_manager.cc:1397] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:29.519411 13042 tablet_bootstrap.cc:654] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:29.519455 13040 raft_consensus.cc:348] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } }
I20260620 14:51:29.519533 13040 raft_consensus.cc:374] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:29.519553 13040 raft_consensus.cc:729] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d9b659fc9ff456c95f4d2228dedf13b, State: Initialized, Role: FOLLOWER
I20260620 14:51:29.519604 13040 consensus_queue.cc:260] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } }
I20260620 14:51:29.519784 13040 ts_tablet_manager.cc:1428] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:29.519945 13042 tablet_bootstrap.cc:492] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d: No bootstrap required, opened a new log
I20260620 14:51:29.519977 13042 ts_tablet_manager.cc:1397] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:29.520090 13042 raft_consensus.cc:348] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } }
I20260620 14:51:29.520102 13045 tablet_bootstrap.cc:492] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68: Bootstrap starting.
I20260620 14:51:29.520148 13042 raft_consensus.cc:374] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:29.520162 13042 raft_consensus.cc:729] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ea2c6b432a49d792f5ebce5457c88d, State: Initialized, Role: FOLLOWER
I20260620 14:51:29.520259 13042 consensus_queue.cc:260] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } }
I20260620 14:51:29.520448 13042 ts_tablet_manager.cc:1428] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:29.520978 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.520826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.130:48015)
0620 14:51:29.520895 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.520934 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:51:29.520993 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.520711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.129:49905)
0620 14:51:29.520931 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.520962 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260620 14:51:29.521271 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.521186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.130:37893)
0620 14:51:29.521226 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.521248 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 14:51:29.521409 12803 heartbeater.cc:503] Master 127.5.118.189:42711 was elected leader, sending a full tablet report...
I20260620 14:51:29.521649 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.521496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.129:49387)
0620 14:51:29.521599 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.521624 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 14:51:29.522038 12666 heartbeater.cc:503] Master 127.5.118.189:42711 was elected leader, sending a full tablet report...
I20260620 14:51:29.522083 13045 tablet_bootstrap.cc:654] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:29.522634 13045 tablet_bootstrap.cc:492] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68: No bootstrap required, opened a new log
I20260620 14:51:29.522676 13045 ts_tablet_manager.cc:1397] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:51:29.522773 13045 raft_consensus.cc:348] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } }
I20260620 14:51:29.522823 13045 raft_consensus.cc:374] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:29.522836 13045 raft_consensus.cc:729] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070974bf9d9b4b6ca3168b73c80bad68, State: Initialized, Role: FOLLOWER
I20260620 14:51:29.522892 13045 consensus_queue.cc:260] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } }
I20260620 14:51:29.523070 13045 ts_tablet_manager.cc:1428] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:29.523326 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.523234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.131:58415)
0620 14:51:29.523269 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.523299 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 14:51:29.523622 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.523532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.131:44177)
0620 14:51:29.523564 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.523592 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 14:51:29.714095 13046 raft_consensus.cc:482] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:29.714195 13046 raft_consensus.cc:504] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } }
I20260620 14:51:29.714521 13046 leader_election.cc:288] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d9b659fc9ff456c95f4d2228dedf13b (127.5.118.129:32795), 070974bf9d9b4b6ca3168b73c80bad68 (127.5.118.131:43601)
I20260620 14:51:29.717682 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.714727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:32795 (local address 127.5.118.130:56521)
0620 14:51:29.714790 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.714802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:29.714856 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:29.715087 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:29.715089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:29.715092 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:29.715181 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.715183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.715881 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.715881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.716836 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.716839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.717405 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.717406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.717485 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.717490 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:51:29.717553 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:51:29.717599 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":47}
I20260620 14:51:29.717693 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.714801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:56521 (local address 127.5.118.129:32795)
0620 14:51:29.714851 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.714853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.714865 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:51:29.714946 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.714947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.714958 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.715072 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:29.715261 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.715785 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.716938 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.717330 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.717387 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.717394 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.717562 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.717599 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:51:29.717971 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.714750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:43601 (local address 127.5.118.130:58007)
0620 14:51:29.714948 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.714959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:29.715019 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:29.715208 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:29.715209 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:29.715210 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:29.715297 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.715299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.715947 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.715948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.716834 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.716837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.717555 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.717556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.717621 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.717626 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:51:29.717694 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:51:29.717892 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":1,"thread_start_us":113,"threads_started":1}
I20260620 14:51:29.718047 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.714825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:58007 (local address 127.5.118.131:43601)
0620 14:51:29.714896 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.714898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.715077 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:51:29.715103 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.715104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.715112 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.715176 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:29.715374 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.715856 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.716941 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.717424 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.717497 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.717508 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.717712 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.717751 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:51:29.717980 12618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93768df37e454690b2faeec227bb8d7d" candidate_uuid: "12ea2c6b432a49d792f5ebce5457c88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" is_pre_election: true
I20260620 14:51:29.718127 12618 raft_consensus.cc:2454] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12ea2c6b432a49d792f5ebce5457c88d in term 0.
I20260620 14:51:29.718338 12690 leader_election.cc:302] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12ea2c6b432a49d792f5ebce5457c88d, 4d9b659fc9ff456c95f4d2228dedf13b; no voters:
I20260620 14:51:29.718364 12891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93768df37e454690b2faeec227bb8d7d" candidate_uuid: "12ea2c6b432a49d792f5ebce5457c88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070974bf9d9b4b6ca3168b73c80bad68" is_pre_election: true
I20260620 14:51:29.718472 12891 raft_consensus.cc:2454] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12ea2c6b432a49d792f5ebce5457c88d in term 0.
I20260620 14:51:29.718475 13046 raft_consensus.cc:2781] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:29.718513 13046 raft_consensus.cc:482] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:29.718547 13046 raft_consensus.cc:3037] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:29.719264 13046 raft_consensus.cc:504] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } }
I20260620 14:51:29.719409 13046 leader_election.cc:288] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [CANDIDATE]: Term 1 election: Requested vote from peers 4d9b659fc9ff456c95f4d2228dedf13b (127.5.118.129:32795), 070974bf9d9b4b6ca3168b73c80bad68 (127.5.118.131:43601)
I20260620 14:51:29.719559 12618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93768df37e454690b2faeec227bb8d7d" candidate_uuid: "12ea2c6b432a49d792f5ebce5457c88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d9b659fc9ff456c95f4d2228dedf13b"
I20260620 14:51:29.719657 12618 raft_consensus.cc:3037] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:29.719650 12891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93768df37e454690b2faeec227bb8d7d" candidate_uuid: "12ea2c6b432a49d792f5ebce5457c88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "070974bf9d9b4b6ca3168b73c80bad68"
I20260620 14:51:29.719756 12891 raft_consensus.cc:3037] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:29.720458 12618 raft_consensus.cc:2454] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12ea2c6b432a49d792f5ebce5457c88d in term 1.
I20260620 14:51:29.720463 12891 raft_consensus.cc:2454] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12ea2c6b432a49d792f5ebce5457c88d in term 1.
I20260620 14:51:29.720616 12688 leader_election.cc:302] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 070974bf9d9b4b6ca3168b73c80bad68, 12ea2c6b432a49d792f5ebce5457c88d; no voters:
I20260620 14:51:29.720750 13046 raft_consensus.cc:2781] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:29.720932 13046 raft_consensus.cc:686] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 1 LEADER]: Becoming Leader. State: Replica: 12ea2c6b432a49d792f5ebce5457c88d, State: Running, Role: LEADER
I20260620 14:51:29.721036 13046 consensus_queue.cc:237] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } } peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } }
I20260620 14:51:29.721716 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.721534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.130:51081)
0620 14:51:29.721649 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.721689 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 14:51:29.721803 12291 catalog_manager.cc:5495] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d reported cstate change: term changed from 0 to 1, leader changed from <none> to 12ea2c6b432a49d792f5ebce5457c88d (127.5.118.130). New cstate: current_term: 1 leader_uuid: "12ea2c6b432a49d792f5ebce5457c88d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12ea2c6b432a49d792f5ebce5457c88d" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 39953 } health_report { overall_health: HEALTHY } } }
I20260620 14:51:29.744246 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.741726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.5.118.130:39953)
0620 14:51:29.741792 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.741794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.741802 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:29.741847 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.741847 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.741865 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.741956 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:29.742104 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.742586 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.743329 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.743557 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.743675 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.743842 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:29.743963 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:29.744066 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.744106 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.744130 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:51:29.745539 12891 raft_consensus.cc:1261] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68 [term 1 FOLLOWER]: Refusing update from remote peer 12ea2c6b432a49d792f5ebce5457c88d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:29.745637 12618 raft_consensus.cc:1261] T 93768df37e454690b2faeec227bb8d7d P 4d9b659fc9ff456c95f4d2228dedf13b [term 1 FOLLOWER]: Refusing update from remote peer 12ea2c6b432a49d792f5ebce5457c88d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:29.745806 13050 consensus_queue.cc:1035] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [LEADER]: Connected to new peer: Peer: permanent_uuid: "070974bf9d9b4b6ca3168b73c80bad68" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:29.745937 13046 consensus_queue.cc:1035] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d9b659fc9ff456c95f4d2228dedf13b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:29.746534 12939 heartbeater.cc:503] Master 127.5.118.189:42711 was elected leader, sending a full tablet report...
I20260620 14:51:29.747004 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.746883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.129:57053)
0620 14:51:29.746929 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.746962 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:51:29.747047 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.746723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.131:39167)
0620 14:51:29.746974 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.747012 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260620 14:51:29.747774 13053 mvcc.cc:205] Tried to move back new op lower bound from 7298937199595245568 to 7298937199497912320. Current Snapshot: MvccSnapshot[applied={T|T < 7298937199595245568}]
I20260620 14:51:29.747779 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.747666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.130:52087)
0620 14:51:29.747713 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.747746 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 14:51:29.747970 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.747757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.131:60649)
0620 14:51:29.747918 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.747945 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 14:51:29.749670 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.749574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.129:39229)
0620 14:51:29.749604 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.749632 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 14:51:29.750459 12292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:29.750584 12292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:29.750687 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.750507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43647 (local address 127.5.118.130:55277)
0620 14:51:29.750631 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.750659 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:43647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260620 14:51:29.752010 12292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 913f34eda30e4e6785bf0c084d487ed8: Sending DeleteTablet for 3 replicas of tablet 93768df37e454690b2faeec227bb8d7d
I20260620 14:51:29.752341 12871 tablet_service.cc:1514] Processing DeleteTablet for tablet 93768df37e454690b2faeec227bb8d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:29 UTC) from {username='oryx', principal='oryx/127.5.118.189@KRBTEST.COM'} at 127.0.0.1:36166
I20260620 14:51:29.752336 12598 tablet_service.cc:1514] Processing DeleteTablet for tablet 93768df37e454690b2faeec227bb8d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:29 UTC) from {username='oryx', principal='oryx/127.5.118.189@KRBTEST.COM'} at 127.0.0.1:47580
I20260620 14:51:29.752578 12735 tablet_service.cc:1514] Processing DeleteTablet for tablet 93768df37e454690b2faeec227bb8d7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:29 UTC) from {username='oryx', principal='oryx/127.5.118.189@KRBTEST.COM'} at 127.0.0.1:36308
I20260620 14:51:29.752758 13065 tablet_replica.cc:330] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d: stopping tablet replica
I20260620 14:51:29.752772 13064 tablet_replica.cc:330] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68: stopping tablet replica
I20260620 14:51:29.752817 13065 raft_consensus.cc:2229] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:29.752827 13064 raft_consensus.cc:2229] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:29.752878 13064 raft_consensus.cc:2258] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:29.752890 13065 raft_consensus.cc:2258] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:29.752926 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 12442
I20260620 14:51:29.753206 13065 ts_tablet_manager.cc:1905] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:29.753693 13064 ts_tablet_manager.cc:1905] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:29.754832 13065 ts_tablet_manager.cc:1918] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:29.754882 13065 log.cc:1196] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/93768df37e454690b2faeec227bb8d7d
I20260620 14:51:29.755121 13065 ts_tablet_manager.cc:1939] T 93768df37e454690b2faeec227bb8d7d P 12ea2c6b432a49d792f5ebce5457c88d: Deleting consensus metadata
I20260620 14:51:29.755431 12276 catalog_manager.cc:4841] TS 12ea2c6b432a49d792f5ebce5457c88d (127.5.118.130:39953): tablet 93768df37e454690b2faeec227bb8d7d (table test-table [id=16a852bf6dd442899058461c54dea1ec]) successfully deleted
I20260620 14:51:29.755558 13064 ts_tablet_manager.cc:1918] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:29.755611 13064 log.cc:1196] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/93768df37e454690b2faeec227bb8d7d
I20260620 14:51:29.755870 13064 ts_tablet_manager.cc:1939] T 93768df37e454690b2faeec227bb8d7d P 070974bf9d9b4b6ca3168b73c80bad68: Deleting consensus metadata
I20260620 14:51:29.756202 12276 catalog_manager.cc:4841] TS 070974bf9d9b4b6ca3168b73c80bad68 (127.5.118.131:43601): tablet 93768df37e454690b2faeec227bb8d7d (table test-table [id=16a852bf6dd442899058461c54dea1ec]) successfully deleted
W20260620 14:51:29.765969 13035 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:29.765822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:32795 (local address 127.0.0.1:47586)
0620 14:51:29.765879 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.765906 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:32795: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 14:51:29.766148 12278 catalog_manager.cc:4584] TS 4d9b659fc9ff456c95f4d2228dedf13b (127.5.118.129:32795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 93768df37e454690b2faeec227bb8d7d: Network error: Client connection negotiation failed: client connection to 127.5.118.129:32795: connect: Connection reset by peer (error 104)
I20260620 14:51:29.766170 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 12672
I20260620 14:51:29.771306 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 12808
I20260620 14:51:29.776142 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 12159
I20260620 14:51:29.781203 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 12346
2026-06-20T14:51:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6605 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:29 dist-test-slave-2m5f krb5kdc[13068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:29 dist-test-slave-2m5f krb5kdc[13068](info): set up 2 sockets
Jun 20 14:51:29 dist-test-slave-2m5f krb5kdc[13068](info): commencing operation
krb5kdc: starting...
W20260620 14:51:30.864975 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:30 dist-test-slave-2m5f krb5kdc[13068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.190.keytab.
Entry for principal oryx/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.190.keytab.
I20260620 14:51:30.988978 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43231
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.190.keytab
--principal=oryx/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:43231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:31.071269 13081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:31.071476 13081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:31.071499 13081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:31.072746 13081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:31.072790 13081 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:51:31.072804 13081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:31.072817 13081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:31.072829 13081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:31.072841 13081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:31.074504 13081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:43231
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.190.keytab
--principal=oryx/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:31.074790 13081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:31.075047 13081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:31.080140 13086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.080152 13087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.080157 13089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:31.080587 13081 server_base.cc:1034] running on GCE node
I20260620 14:51:31.080842 13081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:31.081154 13081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:31.082316 13081 hybrid_clock.cc:648] HybridClock initialized: now 1781967091082295 us; error 37 us; skew 500 ppm
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:31.085003 13081 init.cc:377] Logged in from keytab as oryx/127.5.118.190@KRBTEST.COM (short username oryx)
I20260620 14:51:31.085506 13081 webserver.cc:460] Webserver started at http://127.5.118.190:37651/ using document root <none> and password file <none>
I20260620 14:51:31.085767 13081 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:31.085837 13081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:31.085987 13081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:31.087078 13081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "e749aa343f00450e89ab1c4c6c7fa87a"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "688ece86636b2ececd9924eb88d5250b"
server_key_iv: "db5991b15e00c58be5d008bafdd45e17"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.087447 13081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "e749aa343f00450e89ab1c4c6c7fa87a"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "688ece86636b2ececd9924eb88d5250b"
server_key_iv: "db5991b15e00c58be5d008bafdd45e17"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.088922 13081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:31.089682 13096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.089864 13081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:31.089975 13081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "e749aa343f00450e89ab1c4c6c7fa87a"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "688ece86636b2ececd9924eb88d5250b"
server_key_iv: "db5991b15e00c58be5d008bafdd45e17"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.090061 13081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:31.105065 13081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:31.107242 13081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:31.107419 13081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:31.112493 13081 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:43231
I20260620 14:51:31.112601 13158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:43231 every 8 connection(s)
I20260620 14:51:31.112963 13081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:31.113667 13159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:31.113967 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 13081
I20260620 14:51:31.114082 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:31.114275 5594 external_mini_cluster.cc:1428] Setting key 42a4e4ac494104e4e7b30ec1a2ff0f21
I20260620 14:51:31.117151 13159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a: Bootstrap starting.
I20260620 14:51:31.117947 13159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:31.118284 13159 log.cc:824] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:31.119141 13159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a: No bootstrap required, opened a new log
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
I20260620 14:51:31.121778 13159 raft_consensus.cc:348] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e749aa343f00450e89ab1c4c6c7fa87a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43231 } }
I20260620 14:51:31.121995 13159 raft_consensus.cc:374] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:31.122052 13159 raft_consensus.cc:729] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e749aa343f00450e89ab1c4c6c7fa87a, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.122205 13159 consensus_queue.cc:260] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e749aa343f00450e89ab1c4c6c7fa87a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43231 } }
I20260620 14:51:31.122303 13159 raft_consensus.cc:388] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:31.122350 13159 raft_consensus.cc:482] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:31.122408 13159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:31.123247 13159 raft_consensus.cc:504] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e749aa343f00450e89ab1c4c6c7fa87a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43231 } }
I20260620 14:51:31.123376 13159 leader_election.cc:302] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e749aa343f00450e89ab1c4c6c7fa87a; no voters:
I20260620 14:51:31.123642 13159 leader_election.cc:288] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:31.123705 13164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:31.123859 13164 raft_consensus.cc:686] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [term 1 LEADER]: Becoming Leader. State: Replica: e749aa343f00450e89ab1c4c6c7fa87a, State: Running, Role: LEADER
I20260620 14:51:31.123948 13164 consensus_queue.cc:237] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e749aa343f00450e89ab1c4c6c7fa87a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43231 } }
I20260620 14:51:31.123986 13159 sys_catalog.cc:564] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:31.124310 13165 sys_catalog.cc:455] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e749aa343f00450e89ab1c4c6c7fa87a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e749aa343f00450e89ab1c4c6c7fa87a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43231 } } }
I20260620 14:51:31.124403 13165 sys_catalog.cc:458] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:31.124653 13166 sys_catalog.cc:455] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e749aa343f00450e89ab1c4c6c7fa87a. Latest consensus state: current_term: 1 leader_uuid: "e749aa343f00450e89ab1c4c6c7fa87a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e749aa343f00450e89ab1c4c6c7fa87a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43231 } } }
I20260620 14:51:31.124955 13169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:31.125754 13169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:31.126976 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.114842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.5.118.190:43231)
0620 14:51:31.115061 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.115074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.115116 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:51:31.115685 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.115694 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.115765 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.115953 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.116604 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.118152 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.118929 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.119603 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.119732 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.122482 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.122487 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.122488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.122501 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.124871 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.125455 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.125458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.125459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.125625 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.125881 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.125882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.125883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.126249 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.126456 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.126548 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.126698 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":75,"threads_started":1}
I20260620 14:51:31.126824 13166 sys_catalog.cc:458] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:31.127471 13169 catalog_manager.cc:1347] Generated new cluster ID: d3bd16802f854b2c839238cde9699dc5
I20260620 14:51:31.127527 13169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:31.132732 13169 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:31.133673 13169 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:31.142038 13169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a: Generated new TSK 0
I20260620 14:51:31.142253 13169 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab.
Entry for principal oryx/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab.
I20260620 14:51:31.178803 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43231
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab
--principal=oryx/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:31.266860 13185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:31.267110 13185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:31.267151 13185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:31.268366 13185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:31.268440 13185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:31.268476 13185 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:51:31.268546 13185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:31.270081 13185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab
--principal=oryx/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.118.190:43231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:31.270373 13185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:31.270676 13185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:31.275872 13193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:31.275897 13185 server_base.cc:1034] running on GCE node
W20260620 14:51:31.275843 13191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.275862 13190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:31.276414 13185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:31.276686 13185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:31.277864 13185 hybrid_clock.cc:648] HybridClock initialized: now 1781967091277859 us; error 38 us; skew 500 ppm
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:31.281020 13185 init.cc:377] Logged in from keytab as oryx/127.5.118.129@KRBTEST.COM (short username oryx)
I20260620 14:51:31.281586 13185 webserver.cc:460] Webserver started at http://127.5.118.129:41531/ using document root <none> and password file <none>
I20260620 14:51:31.281827 13185 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:31.281899 13185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:31.282059 13185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:31.283147 13185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "65e2e4b091ee437d8ccdcb7868521132"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "b4223f9c00d665ee2a968486386719f6"
server_key_iv: "c7d36e5e9f491240f032ad8dc42f1037"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.283529 13185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "65e2e4b091ee437d8ccdcb7868521132"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "b4223f9c00d665ee2a968486386719f6"
server_key_iv: "c7d36e5e9f491240f032ad8dc42f1037"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.285003 13185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:31.285960 13200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.286173 13185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:31.286278 13185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "65e2e4b091ee437d8ccdcb7868521132"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "b4223f9c00d665ee2a968486386719f6"
server_key_iv: "c7d36e5e9f491240f032ad8dc42f1037"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.286365 13185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:31.309895 13185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:31.311861 13185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:31.312040 13185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:31.312732 13185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:31.312796 13185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.312831 13185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:31.312858 13185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.319931 13185 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:42167
I20260620 14:51:31.320012 13315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:42167 every 8 connection(s)
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
I20260620 14:51:31.320374 13185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:31.323612 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.313113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:57927 (local address 127.5.118.190:43231)
0620 14:51:31.313188 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.313191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.313415 (+ 224us) server_negotiation.cc:407] Connection header received
0620 14:51:31.313947 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.313957 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.313971 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.314058 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.315101 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.315935 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.317220 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.317452 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.317536 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.321012 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.321013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.321014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.321021 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.322186 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.322550 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.322550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.322551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.322600 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.322905 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.322905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.322906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.323050 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.323104 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.323421 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.323517 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:51:31.323864 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.313113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43231 (local address 127.5.118.129:57927)
0620 14:51:31.313274 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.313313 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.313841 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.314163 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.314166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.314664 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:31.314977 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.314985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.316043 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.316044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.317129 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.317132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.317559 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.317560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.317683 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.318895 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:31.318906 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:31.320858 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:31.322255 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.322261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.322263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.322480 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.322683 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.322684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.322685 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.322812 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.323133 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:31.323139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:31.323261 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.323354 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.323568 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260620 14:51:31.324438 13316 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43231
I20260620 14:51:31.324646 13316 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:31.324636 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 13185
I20260620 14:51:31.324759 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:31.324894 13316 heartbeater.cc:511] Master 127.5.118.190:43231 requested a full tablet report, sending...
I20260620 14:51:31.324999 5594 external_mini_cluster.cc:1428] Setting key 9e0815b62afc4fc400bcaeac124d33dc
I20260620 14:51:31.325464 13112 ts_manager.cc:194] Registered new tserver with Master: 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129:42167)
I20260620 14:51:31.326166 13112 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:57927
I20260620 14:51:31.331931 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.325701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43231 (local address 127.5.118.129:42911)
0620 14:51:31.325830 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.325840 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.325896 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.326028 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.326029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.326218 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:31.326303 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.326304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.326964 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.326965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.327511 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.327513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.328089 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.328090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.328165 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.328662 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:31.328667 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:31.329074 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:31.330884 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.330885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.330886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.331070 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.331296 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.331296 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.331296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.331359 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.331666 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:31.331667 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:31.331708 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.331766 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.331851 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260620 14:51:31.332033 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.325658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:42911 (local address 127.5.118.190:43231)
0620 14:51:31.325706 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.325708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.325883 (+ 175us) server_negotiation.cc:407] Connection header received
0620 14:51:31.325953 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.325954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.325964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.326018 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.326396 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.326870 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.327635 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.327947 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.327994 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.329164 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.329166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.329166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.329174 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.330781 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.331152 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.331154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.331154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.331221 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.331425 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.331426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.331427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.331591 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.331673 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.331813 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.331904 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for oryx/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab.
Entry for principal oryx/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab.
I20260620 14:51:31.351292 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43231
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab
--principal=oryx/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:31.434132 13325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:31.434399 13325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:31.434443 13325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:31.435678 13325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:31.435750 13325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:31.435784 13325 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:51:31.435827 13325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:31.437417 13325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab
--principal=oryx/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.118.190:43231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:31.437711 13325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:31.437958 13325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:51:31.443471 13325 server_base.cc:1034] running on GCE node
W20260620 14:51:31.443441 13333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.443461 13331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.443441 13330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:31.444023 13325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:31.444326 13325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:31.445539 13325 hybrid_clock.cc:648] HybridClock initialized: now 1781967091445530 us; error 40 us; skew 500 ppm
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:31.448127 13325 init.cc:377] Logged in from keytab as oryx/127.5.118.130@KRBTEST.COM (short username oryx)
I20260620 14:51:31.448640 13325 webserver.cc:460] Webserver started at http://127.5.118.130:34705/ using document root <none> and password file <none>
I20260620 14:51:31.448873 13325 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:31.448948 13325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:31.449108 13325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:31.450207 13325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "f50f447f010c40ff8c65dd826506ac8b"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "08c92b96a793031a9d22f12175d5fd45"
server_key_iv: "4104b7998a84571c573d41a8fd8b6652"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.450603 13325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "f50f447f010c40ff8c65dd826506ac8b"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "08c92b96a793031a9d22f12175d5fd45"
server_key_iv: "4104b7998a84571c573d41a8fd8b6652"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.451975 13325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:51:31.452840 13340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.453019 13325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:31.453120 13325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "f50f447f010c40ff8c65dd826506ac8b"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "08c92b96a793031a9d22f12175d5fd45"
server_key_iv: "4104b7998a84571c573d41a8fd8b6652"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.453210 13325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:31.472043 13325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:31.474534 13325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:31.474709 13325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:31.475325 13325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:31.475394 13325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.475441 13325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:31.475469 13325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.483062 13325 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:35843
I20260620 14:51:31.483144 13455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:35843 every 8 connection(s)
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
I20260620 14:51:31.483456 13325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:31.486536 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 13325
I20260620 14:51:31.486668 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:31.486791 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.475965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:35301 (local address 127.5.118.190:43231)
0620 14:51:31.476042 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.476045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.476313 (+ 268us) server_negotiation.cc:407] Connection header received
0620 14:51:31.477051 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.477053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.477066 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.477119 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.477851 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.478786 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.480608 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.480989 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.481069 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.484040 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.484042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.484042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.484049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.485316 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.485751 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.485752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.485752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.485810 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.486094 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.486095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.486096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.486231 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.486310 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.486588 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.486692 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:51:31.486896 5594 external_mini_cluster.cc:1428] Setting key 22e301bc8db92930b708db0b5fffd76f
I20260620 14:51:31.486943 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.475951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43231 (local address 127.5.118.130:35301)
0620 14:51:31.476130 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.476200 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.476897 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.477224 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.477229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.477523 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:31.477735 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.477740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.478911 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.478913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.480485 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.480489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.481113 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.481114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.481278 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.482036 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:31.482049 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:31.483917 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:31.485399 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.485407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.485409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.485657 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.485882 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.485882 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.485883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.485994 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.486292 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:31.486296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:31.486418 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.486501 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.486690 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260620 14:51:31.487571 13456 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43231
I20260620 14:51:31.488286 13456 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:31.488550 13456 heartbeater.cc:511] Master 127.5.118.190:43231 requested a full tablet report, sending...
I20260620 14:51:31.488937 13112 ts_manager.cc:194] Registered new tserver with Master: f50f447f010c40ff8c65dd826506ac8b (127.5.118.130:35843)
I20260620 14:51:31.489387 13112 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:35301
I20260620 14:51:31.495491 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.488681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:57895 (local address 127.5.118.190:43231)
0620 14:51:31.488781 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.488785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.488922 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:51:31.488984 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.488985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.488996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.489051 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.489490 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.490269 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.491401 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.491656 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.491732 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.492892 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.492893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.492893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.492900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.494356 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.494759 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.494760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.494760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.494812 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.495007 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.495008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.495008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.495140 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.495191 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.495369 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.495425 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:51:31.495625 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.488638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43231 (local address 127.5.118.130:57895)
0620 14:51:31.488838 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.488868 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.488933 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.489102 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.489103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.489329 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:31.489401 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.489404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.490402 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.490402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.491315 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.491317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.491750 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.491751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.491839 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.492353 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:31.492358 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:31.492809 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:31.494441 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.494442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.494443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.494676 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.494876 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.494877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.494877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.494941 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.495209 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:31.495210 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:31.495264 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.495355 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.495471 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for oryx/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab.
Entry for principal oryx/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab.
I20260620 14:51:31.515393 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43231
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab
--principal=oryx/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:41635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:31.598446 13465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:31.598708 13465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:31.598750 13465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:31.599969 13465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:31.600049 13465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:31.600085 13465 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:51:31.600121 13465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:31.601756 13465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:41635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab
--principal=oryx/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.118.190:43231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:31.602043 13465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:31.602286 13465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:31.607311 13471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.607337 13473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:31.607338 13470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:31.607457 13465 server_base.cc:1034] running on GCE node
I20260620 14:51:31.607776 13465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:31.608043 13465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:31.609187 13465 hybrid_clock.cc:648] HybridClock initialized: now 1781967091609177 us; error 59 us; skew 500 ppm
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:31.611800 13465 init.cc:377] Logged in from keytab as oryx/127.5.118.131@KRBTEST.COM (short username oryx)
I20260620 14:51:31.612282 13465 webserver.cc:460] Webserver started at http://127.5.118.131:45389/ using document root <none> and password file <none>
I20260620 14:51:31.612480 13465 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:31.612526 13465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:31.612638 13465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:31.613705 13465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "dc83caa8b7c20fe9489712db26816efa"
server_key_iv: "c25e765d702b5a3ad92bf2d56dfadbf6"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.614028 13465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "dc83caa8b7c20fe9489712db26816efa"
server_key_iv: "c25e765d702b5a3ad92bf2d56dfadbf6"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.615343 13465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:51:31.616122 13480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.616340 13465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:31.616434 13465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5"
format_stamp: "Formatted at 2026-06-20 14:51:31 on dist-test-slave-2m5f"
server_key: "dc83caa8b7c20fe9489712db26816efa"
server_key_iv: "c25e765d702b5a3ad92bf2d56dfadbf6"
server_key_version: "encryptionkey@0"
I20260620 14:51:31.616519 13465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:31.625667 13465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:31.628135 13465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:31.628331 13465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:31.628875 13465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:31.628938 13465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.628981 13465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:31.629009 13465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.635998 13465 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:44277
I20260620 14:51:31.636099 13595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:44277 every 8 connection(s)
I20260620 14:51:31.636448 13465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
I20260620 14:51:31.639675 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.629369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:42677 (local address 127.5.118.190:43231)
0620 14:51:31.629441 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.629444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.629641 (+ 197us) server_negotiation.cc:407] Connection header received
0620 14:51:31.630114 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.630117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.630130 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.630214 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.631110 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.631941 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.633805 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.634114 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.634181 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.637092 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.637094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.637094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.637102 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.638308 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.638723 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.638723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.638724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.638775 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.639002 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.639003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.639003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.639106 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.639158 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.639428 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.639568 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:51:31.639689 13489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.629342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43231 (local address 127.5.118.131:42677)
0620 14:51:31.629479 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.629549 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.630017 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.630317 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.630333 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.630705 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:31.630997 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.631004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.632081 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.632083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.633659 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.633663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.634223 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.634224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.634397 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.635080 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:31.635093 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:31.636949 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:31.638407 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.638414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.638416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.638647 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.638837 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.638838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.638838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.638945 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.639180 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:31.639185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:31.639306 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.639398 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.639529 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260620 14:51:31.640465 13596 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43231
I20260620 14:51:31.640625 13596 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:31.640646 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 13465
I20260620 14:51:31.640748 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:31.640827 13596 heartbeater.cc:511] Master 127.5.118.190:43231 requested a full tablet report, sending...
I20260620 14:51:31.640971 5594 external_mini_cluster.cc:1428] Setting key f6a9e0829de825c362bd38f10cab44d0
I20260620 14:51:31.641212 13112 ts_manager.cc:194] Registered new tserver with Master: ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277)
I20260620 14:51:31.641723 13112 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:42677
I20260620 14:51:31.641884 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:51:31.649986 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.641489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43231 (local address 127.5.118.131:33713)
0620 14:51:31.641611 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.641808 (+ 197us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.643556 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.643744 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.643744 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.643895 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:31.643947 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.643949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.644904 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.644905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.645456 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.645459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.645987 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.645988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.646045 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.646607 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:31.646612 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:31.646908 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:31.649044 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.649045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.649045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.649216 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.649434 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.649435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.649435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.649486 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.649742 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:31.649743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:31.649780 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.649832 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.649914 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20260620 14:51:31.650094 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.641643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:33713 (local address 127.5.118.190:43231)
0620 14:51:31.641676 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.641677 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.641856 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:51:31.643648 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.643650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.643662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.643733 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.644018 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.644776 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.645550 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.645868 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.645952 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.647033 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.647035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.647035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.647043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.648926 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.649299 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.649300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.649300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.649366 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.649531 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.649532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.649532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.649676 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.649750 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.649878 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.649966 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:51:31.665418 13162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:31.659505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.5.118.190:43231)
0620 14:51:31.659582 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.659584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.659609 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:51:31.659633 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.659634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.659643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.659686 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.660324 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.660832 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.661616 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.661848 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.661918 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.665325 (+ 3407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53530: BlockingRecv error: recv got EOF from 127.0.0.1:53530 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
I20260620 14:51:31.673672 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.666325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.5.118.190:43231)
0620 14:51:31.666394 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.666396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.666490 (+ 94us) server_negotiation.cc:407] Connection header received
0620 14:51:31.666598 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.666599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.666616 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.666683 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.667245 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.667780 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.668651 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.668871 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.668963 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.671198 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.671200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.671200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.671209 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.672513 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.672855 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.672856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.672857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.672914 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.673115 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.673116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.673116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.673226 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.673335 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.673460 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.673559 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:51:31.675477 13112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:31.677220 13112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:31.684672 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.679816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:42167 (local address 127.0.0.1:37096)
0620 14:51:31.680353 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.680362 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.680402 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.680925 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.680926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.680927 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:31.680976 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.680978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.682233 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.682233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.683496 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.683499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.684343 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.684344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.684430 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.684446 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.684516 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.684569 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":146,"threads_started":1}
I20260620 14:51:31.684785 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.679791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:44277 (local address 127.0.0.1:48740)
0620 14:51:31.680030 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.680063 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.680149 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.680663 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.680666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.680669 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:31.680749 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.680750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.682147 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.682148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.683481 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.683484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.684526 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.684527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.684579 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.684581 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.684638 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.684671 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":77,"thread_start_us":61,"threads_started":1}
I20260620 14:51:31.685045 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.680147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.5.118.129:42167)
0620 14:51:31.680760 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.680765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.680774 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:31.680807 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.680810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.680824 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.680923 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:31.681069 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.682157 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.683591 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.684239 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.684342 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.684433 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.684921 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.684962 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":127,"threads_started":1}
I20260620 14:51:31.685077 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.679816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:35843 (local address 127.0.0.1:59814)
0620 14:51:31.680365 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.680378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.680434 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.681202 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.681202 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.681205 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:31.681256 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.681257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.682120 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.682120 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.683675 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.683677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.684834 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.684835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.684916 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.684920 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.684968 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.685008 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":161,"thread_start_us":148,"threads_started":1}
I20260620 14:51:31.685345 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.680155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.5.118.130:35843)
0620 14:51:31.681048 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.681053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.681063 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:31.681104 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.681105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.681116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.681202 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:31.681313 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.681994 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.683769 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.684665 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.684729 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.684792 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.685184 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.685229 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":127,"threads_started":1}
I20260620 14:51:31.685499 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.680149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.5.118.131:44277)
0620 14:51:31.680397 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.680402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.680496 (+ 94us) server_negotiation.cc:407] Connection header received
0620 14:51:31.680542 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.680544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.680559 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.680634 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:31.680899 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.682049 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.683609 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.684434 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.684524 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.684599 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.684658 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.684705 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":127,"threads_started":1}
I20260620 14:51:31.685765 13250 tablet_service.cc:1467] Processing CreateTablet for tablet cf43deb1a1a244f3a8ce7172047ab3f8 (DEFAULT_TABLE table=test-table [id=d230cae93dc845dfbf5dfb0e6d1c713d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:31.686177 13250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf43deb1a1a244f3a8ce7172047ab3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:31.686178 13390 tablet_service.cc:1467] Processing CreateTablet for tablet cf43deb1a1a244f3a8ce7172047ab3f8 (DEFAULT_TABLE table=test-table [id=d230cae93dc845dfbf5dfb0e6d1c713d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:31.686517 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf43deb1a1a244f3a8ce7172047ab3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:31.689759 13624 tablet_bootstrap.cc:492] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: Bootstrap starting.
I20260620 14:51:31.689757 13625 tablet_bootstrap.cc:492] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132: Bootstrap starting.
I20260620 14:51:31.690440 13624 tablet_bootstrap.cc:654] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:31.690469 13625 tablet_bootstrap.cc:654] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:31.690452 13530 tablet_service.cc:1467] Processing CreateTablet for tablet cf43deb1a1a244f3a8ce7172047ab3f8 (DEFAULT_TABLE table=test-table [id=d230cae93dc845dfbf5dfb0e6d1c713d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:31.690799 13625 log.cc:824] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:31.690807 13624 log.cc:824] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:31.690832 13530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf43deb1a1a244f3a8ce7172047ab3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:31.691632 13624 tablet_bootstrap.cc:492] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: No bootstrap required, opened a new log
I20260620 14:51:31.691701 13624 ts_tablet_manager.cc:1397] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:31.691835 13625 tablet_bootstrap.cc:492] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132: No bootstrap required, opened a new log
I20260620 14:51:31.691968 13625 ts_tablet_manager.cc:1397] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:51:31.693593 13625 raft_consensus.cc:348] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } }
I20260620 14:51:31.693749 13625 raft_consensus.cc:374] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:31.693792 13625 raft_consensus.cc:729] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e2e4b091ee437d8ccdcb7868521132, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.693812 13624 raft_consensus.cc:348] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } }
I20260620 14:51:31.693928 13625 consensus_queue.cc:260] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } }
I20260620 14:51:31.694001 13624 raft_consensus.cc:374] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:31.694037 13624 raft_consensus.cc:729] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50f447f010c40ff8c65dd826506ac8b, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.694154 13624 consensus_queue.cc:260] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } }
I20260620 14:51:31.694214 13625 ts_tablet_manager.cc:1428] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:51:31.694350 13316 heartbeater.cc:503] Master 127.5.118.190:43231 was elected leader, sending a full tablet report...
I20260620 14:51:31.694442 13624 ts_tablet_manager.cc:1428] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:51:31.694629 13456 heartbeater.cc:503] Master 127.5.118.190:43231 was elected leader, sending a full tablet report...
I20260620 14:51:31.694626 13630 tablet_bootstrap.cc:492] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Bootstrap starting.
I20260620 14:51:31.695343 13630 tablet_bootstrap.cc:654] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:31.695701 13630 log.cc:824] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:31.696533 13630 tablet_bootstrap.cc:492] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: No bootstrap required, opened a new log
I20260620 14:51:31.696619 13630 ts_tablet_manager.cc:1397] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:31.698007 13630 raft_consensus.cc:348] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } }
I20260620 14:51:31.698164 13630 raft_consensus.cc:374] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:31.698211 13630 raft_consensus.cc:729] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba67d8f9c7ad4828b5adf0dc8c8b36c5, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.698335 13630 consensus_queue.cc:260] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } }
I20260620 14:51:31.698592 13630 ts_tablet_manager.cc:1428] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:51:31.698668 13596 heartbeater.cc:503] Master 127.5.118.190:43231 was elected leader, sending a full tablet report...
I20260620 14:51:31.722285 13628 raft_consensus.cc:482] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:31.722522 13628 raft_consensus.cc:504] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } }
I20260620 14:51:31.722841 13628 leader_election.cc:288] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277), f50f447f010c40ff8c65dd826506ac8b (127.5.118.130:35843)
I20260620 14:51:31.726845 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.723167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:54245 (local address 127.5.118.131:44277)
0620 14:51:31.723212 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.723214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.723425 (+ 211us) server_negotiation.cc:407] Connection header received
0620 14:51:31.723455 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.723456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.723466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.723531 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:31.723690 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.724624 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.725863 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.726481 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.726555 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.726564 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.726727 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.726765 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:51:31.726835 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.723160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:45143 (local address 127.5.118.130:35843)
0620 14:51:31.723212 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.723214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.723225 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:51:31.723265 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.723266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.723275 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.723463 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:31.723690 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.724579 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.725849 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.726456 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.726529 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.726539 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.726713 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.726751 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:51:31.726892 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.723061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:35843 (local address 127.5.118.129:45143)
0620 14:51:31.723111 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.723124 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.723174 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.723472 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.723474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.723476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:31.723596 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.723597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.724703 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.724705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.725749 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.725752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.726527 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.726528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.726627 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.726630 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.726695 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.726747 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 14:51:31.726912 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.723064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:44277 (local address 127.5.118.129:54245)
0620 14:51:31.723305 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.723313 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.723398 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.723529 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.723530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.723531 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:31.723595 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.723597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.724716 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.724717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.725737 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.725740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.726554 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.726555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.726642 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.726645 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.726721 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.726767 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":76,"thread_start_us":112,"threads_started":1}
I20260620 14:51:31.727221 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf43deb1a1a244f3a8ce7172047ab3f8" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50f447f010c40ff8c65dd826506ac8b" is_pre_election: true
I20260620 14:51:31.727425 13410 raft_consensus.cc:2454] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 0.
I20260620 14:51:31.727437 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf43deb1a1a244f3a8ce7172047ab3f8" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" is_pre_election: true
I20260620 14:51:31.727612 13550 raft_consensus.cc:2454] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 0.
I20260620 14:51:31.727697 13201 leader_election.cc:302] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e2e4b091ee437d8ccdcb7868521132, f50f447f010c40ff8c65dd826506ac8b; no voters:
I20260620 14:51:31.727840 13628 raft_consensus.cc:2781] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:31.727890 13628 raft_consensus.cc:482] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:31.727916 13628 raft_consensus.cc:3037] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:31.728873 13628 raft_consensus.cc:504] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } }
I20260620 14:51:31.729045 13628 leader_election.cc:288] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 election: Requested vote from peers ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277), f50f447f010c40ff8c65dd826506ac8b (127.5.118.130:35843)
I20260620 14:51:31.729195 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf43deb1a1a244f3a8ce7172047ab3f8" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50f447f010c40ff8c65dd826506ac8b"
I20260620 14:51:31.729252 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf43deb1a1a244f3a8ce7172047ab3f8" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5"
I20260620 14:51:31.729303 13410 raft_consensus.cc:3037] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:31.729369 13550 raft_consensus.cc:3037] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:31.730067 13410 raft_consensus.cc:2454] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 1.
I20260620 14:51:31.730121 13550 raft_consensus.cc:2454] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 1.
I20260620 14:51:31.730245 13201 leader_election.cc:302] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e2e4b091ee437d8ccdcb7868521132, f50f447f010c40ff8c65dd826506ac8b; no voters:
I20260620 14:51:31.730347 13628 raft_consensus.cc:2781] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:31.730532 13628 raft_consensus.cc:686] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 LEADER]: Becoming Leader. State: Replica: 65e2e4b091ee437d8ccdcb7868521132, State: Running, Role: LEADER
I20260620 14:51:31.730641 13628 consensus_queue.cc:237] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } }
I20260620 14:51:31.731529 13112 catalog_manager.cc:5495] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129). New cstate: current_term: 1 leader_uuid: "65e2e4b091ee437d8ccdcb7868521132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } health_report { overall_health: HEALTHY } } }
W20260620 14:51:31.734109 13457 tablet.cc:2369] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:31.743016 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.740665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.5.118.129:42167)
0620 14:51:31.740728 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.740730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.740739 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:31.740774 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.740775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.740792 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.740855 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:31.740987 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.741488 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.742233 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.742423 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.742535 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.742698 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:31.742801 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:31.742867 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.742901 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.742918 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:51:31.745142 13550 raft_consensus.cc:1261] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Refusing update from remote peer 65e2e4b091ee437d8ccdcb7868521132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:31.745203 13410 raft_consensus.cc:1261] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Refusing update from remote peer 65e2e4b091ee437d8ccdcb7868521132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:31.745528 13634 consensus_queue.cc:1035] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:31.745677 13628 consensus_queue.cc:1035] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:31.748039 13637 mvcc.cc:205] Tried to move back new op lower bound from 7298937207784939520 to 7298937207729541120. Current Snapshot: MvccSnapshot[applied={T|T < 7298937207784939520}]
I20260620 14:51:31.750548 13112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:31.750651 13112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:31.751521 13112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a: Sending DeleteTablet for 3 replicas of tablet cf43deb1a1a244f3a8ce7172047ab3f8
I20260620 14:51:31.751847 13530 tablet_service.cc:1514] Processing DeleteTablet for tablet cf43deb1a1a244f3a8ce7172047ab3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:31 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:48740
I20260620 14:51:31.751924 13390 tablet_service.cc:1514] Processing DeleteTablet for tablet cf43deb1a1a244f3a8ce7172047ab3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:31 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:59814
I20260620 14:51:31.752121 13250 tablet_service.cc:1514] Processing DeleteTablet for tablet cf43deb1a1a244f3a8ce7172047ab3f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:31 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37096
I20260620 14:51:31.752368 13648 tablet_replica.cc:330] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132: stopping tablet replica
I20260620 14:51:31.752383 13646 tablet_replica.cc:330] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: stopping tablet replica
I20260620 14:51:31.752388 13112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:51:31.752486 13646 raft_consensus.cc:2229] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 14:51:31.752529 13112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:31.752597 13646 raft_consensus.cc:2258] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:31.752734 13648 raft_consensus.cc:2229] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:31.752887 13648 raft_consensus.cc:2258] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:31.753062 13646 ts_tablet_manager.cc:1905] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:31.753424 13648 ts_tablet_manager.cc:1905] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:31.753708 13530 tablet_service.cc:1467] Processing CreateTablet for tablet af28d5bca1ee48ac94b88e3d5579d53c (DEFAULT_TABLE table=test-table [id=76417b53e576406c907fd7680f42fff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:31.753871 13530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af28d5bca1ee48ac94b88e3d5579d53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:31.755023 13630 tablet_bootstrap.cc:492] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Bootstrap starting.
I20260620 14:51:31.755225 13646 ts_tablet_manager.cc:1918] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:31.755299 13646 log.cc:1196] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/cf43deb1a1a244f3a8ce7172047ab3f8
I20260620 14:51:31.755427 13648 ts_tablet_manager.cc:1918] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:31.755441 13630 tablet_bootstrap.cc:654] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:31.755560 13648 log.cc:1196] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/wal/wals/cf43deb1a1a244f3a8ce7172047ab3f8
I20260620 14:51:31.755577 13646 ts_tablet_manager.cc:1939] T cf43deb1a1a244f3a8ce7172047ab3f8 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting consensus metadata
I20260620 14:51:31.755913 13648 ts_tablet_manager.cc:1939] T cf43deb1a1a244f3a8ce7172047ab3f8 P 65e2e4b091ee437d8ccdcb7868521132: Deleting consensus metadata
I20260620 14:51:31.755987 13250 tablet_service.cc:1467] Processing CreateTablet for tablet af28d5bca1ee48ac94b88e3d5579d53c (DEFAULT_TABLE table=test-table [id=76417b53e576406c907fd7680f42fff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:31.756058 13390 tablet_service.cc:1467] Processing CreateTablet for tablet af28d5bca1ee48ac94b88e3d5579d53c (DEFAULT_TABLE table=test-table [id=76417b53e576406c907fd7680f42fff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:31.756124 13250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af28d5bca1ee48ac94b88e3d5579d53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:31.756191 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af28d5bca1ee48ac94b88e3d5579d53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:31.756398 13100 catalog_manager.cc:4841] TS ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277): tablet cf43deb1a1a244f3a8ce7172047ab3f8 (table test-table [id=d230cae93dc845dfbf5dfb0e6d1c713d]) successfully deleted
I20260620 14:51:31.756776 13630 tablet_bootstrap.cc:492] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5: No bootstrap required, opened a new log
I20260620 14:51:31.756817 13630 ts_tablet_manager.cc:1397] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:51:31.756965 13630 raft_consensus.cc:348] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.757047 13630 raft_consensus.cc:374] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:31.757067 13630 raft_consensus.cc:729] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba67d8f9c7ad4828b5adf0dc8c8b36c5, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.757115 13630 consensus_queue.cc:260] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.757203 13630 ts_tablet_manager.cc:1428] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:31.757421 13625 tablet_bootstrap.cc:492] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132: Bootstrap starting.
I20260620 14:51:31.757825 13625 tablet_bootstrap.cc:654] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:31.757930 13624 tablet_bootstrap.cc:492] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b: Bootstrap starting.
I20260620 14:51:31.758291 13647 tablet_replica.cc:330] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: stopping tablet replica
I20260620 14:51:31.758373 13624 tablet_bootstrap.cc:654] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:31.758404 13647 raft_consensus.cc:2229] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:31.758546 13647 raft_consensus.cc:2258] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:31.758955 13624 tablet_bootstrap.cc:492] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b: No bootstrap required, opened a new log
I20260620 14:51:31.758998 13624 ts_tablet_manager.cc:1397] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:31.758982 13099 catalog_manager.cc:4841] TS 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129:42167): tablet cf43deb1a1a244f3a8ce7172047ab3f8 (table test-table [id=d230cae93dc845dfbf5dfb0e6d1c713d]) successfully deleted
I20260620 14:51:31.759034 13647 ts_tablet_manager.cc:1905] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:31.759136 13624 raft_consensus.cc:348] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.759217 13624 raft_consensus.cc:374] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:31.759243 13624 raft_consensus.cc:729] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50f447f010c40ff8c65dd826506ac8b, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.759285 13624 consensus_queue.cc:260] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.759382 13624 ts_tablet_manager.cc:1428] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:31.759629 13625 tablet_bootstrap.cc:492] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132: No bootstrap required, opened a new log
I20260620 14:51:31.759675 13625 ts_tablet_manager.cc:1397] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:31.759821 13625 raft_consensus.cc:348] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.759882 13625 raft_consensus.cc:374] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:31.759907 13625 raft_consensus.cc:729] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e2e4b091ee437d8ccdcb7868521132, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.759967 13625 consensus_queue.cc:260] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.760061 13625 ts_tablet_manager.cc:1428] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.760679 13647 ts_tablet_manager.cc:1918] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:31.760757 13647 log.cc:1196] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/cf43deb1a1a244f3a8ce7172047ab3f8
I20260620 14:51:31.761027 13647 ts_tablet_manager.cc:1939] T cf43deb1a1a244f3a8ce7172047ab3f8 P f50f447f010c40ff8c65dd826506ac8b: Deleting consensus metadata
I20260620 14:51:31.761364 13097 catalog_manager.cc:4841] TS f50f447f010c40ff8c65dd826506ac8b (127.5.118.130:35843): tablet cf43deb1a1a244f3a8ce7172047ab3f8 (table test-table [id=d230cae93dc845dfbf5dfb0e6d1c713d]) successfully deleted
I20260620 14:51:31.810693 13628 raft_consensus.cc:482] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:31.810793 13628 raft_consensus.cc:504] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.810968 13628 leader_election.cc:288] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f50f447f010c40ff8c65dd826506ac8b (127.5.118.130:35843), ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277)
I20260620 14:51:31.811185 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af28d5bca1ee48ac94b88e3d5579d53c" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50f447f010c40ff8c65dd826506ac8b" is_pre_election: true
I20260620 14:51:31.811182 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af28d5bca1ee48ac94b88e3d5579d53c" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" is_pre_election: true
I20260620 14:51:31.811285 13550 raft_consensus.cc:2454] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 0.
I20260620 14:51:31.811296 13410 raft_consensus.cc:2454] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 0.
I20260620 14:51:31.811482 13204 leader_election.cc:302] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e2e4b091ee437d8ccdcb7868521132, ba67d8f9c7ad4828b5adf0dc8c8b36c5; no voters:
I20260620 14:51:31.811627 13628 raft_consensus.cc:2781] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:31.811666 13628 raft_consensus.cc:482] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:31.811692 13628 raft_consensus.cc:3037] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:31.812423 13628 raft_consensus.cc:504] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.812568 13628 leader_election.cc:288] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 election: Requested vote from peers f50f447f010c40ff8c65dd826506ac8b (127.5.118.130:35843), ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277)
I20260620 14:51:31.812721 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af28d5bca1ee48ac94b88e3d5579d53c" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50f447f010c40ff8c65dd826506ac8b"
I20260620 14:51:31.812732 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af28d5bca1ee48ac94b88e3d5579d53c" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5"
I20260620 14:51:31.812803 13550 raft_consensus.cc:3037] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:31.812803 13410 raft_consensus.cc:3037] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:31.813545 13410 raft_consensus.cc:2454] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 1.
I20260620 14:51:31.813586 13550 raft_consensus.cc:2454] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 1.
I20260620 14:51:31.813722 13201 leader_election.cc:302] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e2e4b091ee437d8ccdcb7868521132, f50f447f010c40ff8c65dd826506ac8b; no voters:
I20260620 14:51:31.813855 13628 raft_consensus.cc:2781] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:31.813939 13628 raft_consensus.cc:686] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 1 LEADER]: Becoming Leader. State: Replica: 65e2e4b091ee437d8ccdcb7868521132, State: Running, Role: LEADER
I20260620 14:51:31.814000 13628 consensus_queue.cc:237] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.814625 13112 catalog_manager.cc:5495] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129). New cstate: current_term: 1 leader_uuid: "65e2e4b091ee437d8ccdcb7868521132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } health_report { overall_health: UNKNOWN } } }
W20260620 14:51:31.821103 13317 tablet.cc:2369] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:51:31 dist-test-slave-2m5f krb5kdc[13068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967091, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.190@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
I20260620 14:51:31.845631 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.837068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43231 (local address 127.0.0.1:53548)
0620 14:51:31.837224 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:31.837235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:31.837328 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:31.837467 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:31.837468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:31.837797 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:31.837843 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.837846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.838830 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.838831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.839358 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:31.839360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.839858 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.839860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:31.839923 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.840887 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:31.840893 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:31.842472 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:31.844414 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.844424 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.844426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.844652 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.844914 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:31.844915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:31.844915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:31.844979 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:31.845314 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:31.845320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:31.845375 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 14:51:31.845431 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:51:31.845552 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260620 14:51:31.845747 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:31.837126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.5.118.190:43231)
0620 14:51:31.837187 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:31.837190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:31.837308 (+ 118us) server_negotiation.cc:407] Connection header received
0620 14:51:31.837391 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:31.837392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:31.837402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:31.837459 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:31.837973 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.838727 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.839449 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:31.839736 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:31.839848 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:31.842581 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:31.842583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:31.842583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:31.842592 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:31.844307 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.844742 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.844743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.844744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.844814 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:31.845082 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:31.845083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:31.845084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:31.845238 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:31.845318 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:31.845487 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 14:51:31.845632 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":34}
I20260620 14:51:31.848464 13112 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:53548:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:51:31.848716 13112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:31.850296 13250 tablet_service.cc:1467] Processing CreateTablet for tablet 14b4146190cb4f4eb7bfa564f09bac85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15ed53753bb44c93a31a9b0f9d65bfd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:51:31.850315 13390 tablet_service.cc:1467] Processing CreateTablet for tablet 14b4146190cb4f4eb7bfa564f09bac85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15ed53753bb44c93a31a9b0f9d65bfd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:51:31.850416 13530 tablet_service.cc:1467] Processing CreateTablet for tablet 14b4146190cb4f4eb7bfa564f09bac85 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15ed53753bb44c93a31a9b0f9d65bfd1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:51:31.850482 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b4146190cb4f4eb7bfa564f09bac85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:31.850482 13250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b4146190cb4f4eb7bfa564f09bac85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:31.850565 13530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b4146190cb4f4eb7bfa564f09bac85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:31.851969 13625 tablet_bootstrap.cc:492] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: Bootstrap starting.
I20260620 14:51:31.852234 13630 tablet_bootstrap.cc:492] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Bootstrap starting.
I20260620 14:51:31.852471 13625 tablet_bootstrap.cc:654] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:31.852666 13630 tablet_bootstrap.cc:654] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:31.852838 13624 tablet_bootstrap.cc:492] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b: Bootstrap starting.
I20260620 14:51:31.853199 13625 tablet_bootstrap.cc:492] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: No bootstrap required, opened a new log
I20260620 14:51:31.853226 13624 tablet_bootstrap.cc:654] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:31.853245 13625 ts_tablet_manager.cc:1397] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:31.853405 13625 raft_consensus.cc:348] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.853493 13625 raft_consensus.cc:374] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:31.853520 13625 raft_consensus.cc:729] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e2e4b091ee437d8ccdcb7868521132, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.853577 13625 consensus_queue.cc:260] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.853681 13625 ts_tablet_manager.cc:1428] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:31.853781 13628 tablet_replica.cc:439] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } }
I20260620 14:51:31.853915 13628 tablet_replica.cc:442] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:31.854167 13630 tablet_bootstrap.cc:492] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: No bootstrap required, opened a new log
I20260620 14:51:31.854215 13630 ts_tablet_manager.cc:1397] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:31.854362 13630 raft_consensus.cc:348] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.854513 13630 raft_consensus.cc:374] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:31.854542 13630 raft_consensus.cc:729] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba67d8f9c7ad4828b5adf0dc8c8b36c5, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.854591 13630 consensus_queue.cc:260] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.854686 13630 ts_tablet_manager.cc:1428] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.854779 13632 tablet_replica.cc:439] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } }
I20260620 14:51:31.854892 13624 tablet_bootstrap.cc:492] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b: No bootstrap required, opened a new log
I20260620 14:51:31.854924 13624 ts_tablet_manager.cc:1397] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:51:31.854909 13632 tablet_replica.cc:442] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:31.855063 13624 raft_consensus.cc:348] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.855137 13624 raft_consensus.cc:374] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:31.855159 13624 raft_consensus.cc:729] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50f447f010c40ff8c65dd826506ac8b, State: Initialized, Role: FOLLOWER
I20260620 14:51:31.855207 13624 consensus_queue.cc:260] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.855301 13624 ts_tablet_manager.cc:1428] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:31.855396 13629 tablet_replica.cc:439] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } }
I20260620 14:51:31.855484 13629 tablet_replica.cc:442] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 14:51:31.887168 13597 tablet.cc:2369] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:31.983742 13628 raft_consensus.cc:482] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:31.983851 13628 raft_consensus.cc:504] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.984067 13628 leader_election.cc:288] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f50f447f010c40ff8c65dd826506ac8b (127.5.118.130:35843), ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277)
I20260620 14:51:31.984350 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b4146190cb4f4eb7bfa564f09bac85" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" is_pre_election: true
I20260620 14:51:31.984369 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b4146190cb4f4eb7bfa564f09bac85" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50f447f010c40ff8c65dd826506ac8b" is_pre_election: true
I20260620 14:51:31.984457 13410 raft_consensus.cc:2454] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 0.
I20260620 14:51:31.984457 13550 raft_consensus.cc:2454] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 0.
I20260620 14:51:31.984661 13204 leader_election.cc:302] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e2e4b091ee437d8ccdcb7868521132, ba67d8f9c7ad4828b5adf0dc8c8b36c5; no voters:
I20260620 14:51:31.984817 13628 raft_consensus.cc:2781] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:31.984869 13628 raft_consensus.cc:482] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:31.984918 13628 raft_consensus.cc:3037] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:31.985574 13628 raft_consensus.cc:504] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.985716 13628 leader_election.cc:288] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 election: Requested vote from peers f50f447f010c40ff8c65dd826506ac8b (127.5.118.130:35843), ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277)
I20260620 14:51:31.985867 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b4146190cb4f4eb7bfa564f09bac85" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50f447f010c40ff8c65dd826506ac8b"
I20260620 14:51:31.985942 13410 raft_consensus.cc:3037] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:31.985890 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14b4146190cb4f4eb7bfa564f09bac85" candidate_uuid: "65e2e4b091ee437d8ccdcb7868521132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5"
I20260620 14:51:31.985988 13550 raft_consensus.cc:3037] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:31.986527 13410 raft_consensus.cc:2454] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 1.
I20260620 14:51:31.986662 13550 raft_consensus.cc:2454] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65e2e4b091ee437d8ccdcb7868521132 in term 1.
I20260620 14:51:31.986707 13201 leader_election.cc:302] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e2e4b091ee437d8ccdcb7868521132, f50f447f010c40ff8c65dd826506ac8b; no voters:
I20260620 14:51:31.986783 13628 raft_consensus.cc:2781] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:31.986831 13628 raft_consensus.cc:686] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 LEADER]: Becoming Leader. State: Replica: 65e2e4b091ee437d8ccdcb7868521132, State: Running, Role: LEADER
I20260620 14:51:31.986882 13628 consensus_queue.cc:237] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:31.987238 13634 tablet_replica.cc:439] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: TxnStatusTablet state changed. Reason: New leader 65e2e4b091ee437d8ccdcb7868521132. Latest consensus state: current_term: 1 leader_uuid: "65e2e4b091ee437d8ccdcb7868521132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } }
I20260620 14:51:31.987349 13634 tablet_replica.cc:442] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:51:31.987535 13663 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:51:31.987684 13663 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:51:31.987630 13112 catalog_manager.cc:5495] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129). New cstate: current_term: 1 leader_uuid: "65e2e4b091ee437d8ccdcb7868521132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:32.004513 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.001738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:42167 (local address 127.0.0.1:37122)
0620 14:51:32.001820 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.001834 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.001896 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.002046 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.002047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.002048 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:51:32.002095 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.002097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.002987 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.002987 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.003565 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.003566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.004052 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.004053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.004127 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.004143 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:51:32.004364 (+ 221us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:51:32.004371 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.004407 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.004450 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:51:32.004526 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.001839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.5.118.129:42167)
0620 14:51:32.001907 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.001909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.001921 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:51:32.001957 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.001958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.001971 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.002029 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:32.002150 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.002916 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.003647 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.003966 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.004037 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.004267 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:32.004303 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:32.004357 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.004411 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.004436 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:51:32.005648 13550 raft_consensus.cc:1261] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Refusing update from remote peer 65e2e4b091ee437d8ccdcb7868521132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:32.005699 13410 raft_consensus.cc:1261] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Refusing update from remote peer 65e2e4b091ee437d8ccdcb7868521132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:32.005882 13634 consensus_queue.cc:1035] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:32.005981 13628 consensus_queue.cc:1035] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:32.006407 13629 tablet_replica.cc:439] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b: TxnStatusTablet state changed. Reason: New leader 65e2e4b091ee437d8ccdcb7868521132. Latest consensus state: current_term: 1 leader_uuid: "65e2e4b091ee437d8ccdcb7868521132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } }
I20260620 14:51:32.006508 13629 tablet_replica.cc:442] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:32.006505 13632 tablet_replica.cc:439] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: TxnStatusTablet state changed. Reason: New leader 65e2e4b091ee437d8ccdcb7868521132. Latest consensus state: current_term: 1 leader_uuid: "65e2e4b091ee437d8ccdcb7868521132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } }
I20260620 14:51:32.006608 13632 tablet_replica.cc:442] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:32.007270 13634 tablet_replica.cc:439] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65e2e4b091ee437d8ccdcb7868521132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } }
I20260620 14:51:32.007357 13634 tablet_replica.cc:442] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:51:32.007421 13639 mvcc.cc:205] Tried to move back new op lower bound from 7298937208853340160 to 7298937208778854400. Current Snapshot: MvccSnapshot[applied={T|T < 7298937208853340160}]
I20260620 14:51:32.007517 13632 tablet_replica.cc:439] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65e2e4b091ee437d8ccdcb7868521132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } }
I20260620 14:51:32.007584 13632 tablet_replica.cc:442] T 14b4146190cb4f4eb7bfa564f09bac85 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:32.007553 13634 tablet_replica.cc:439] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65e2e4b091ee437d8ccdcb7868521132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } }
I20260620 14:51:32.007624 13634 tablet_replica.cc:442] T 14b4146190cb4f4eb7bfa564f09bac85 P 65e2e4b091ee437d8ccdcb7868521132: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:51:32.008371 13629 tablet_replica.cc:439] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65e2e4b091ee437d8ccdcb7868521132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } } }
I20260620 14:51:32.008459 13629 tablet_replica.cc:442] T 14b4146190cb4f4eb7bfa564f09bac85 P f50f447f010c40ff8c65dd826506ac8b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:51:32.013787 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.011267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:57375 (local address 127.5.118.129:42167)
0620 14:51:32.011327 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.011330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.011402 (+ 72us) server_negotiation.cc:407] Connection header received
0620 14:51:32.011478 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.011479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.011488 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.011543 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:32.011650 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.012128 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.013027 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.013210 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.013292 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.013456 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:32.013513 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:32.013578 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.013636 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.013679 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260620 14:51:32.013789 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.011140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:42167 (local address 127.5.118.129:57375)
0620 14:51:32.011343 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.011354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.011419 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.011547 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.011547 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.011549 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:51:32.011602 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.011604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.012248 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.012249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.012933 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.012936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.013292 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.013293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.013341 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.013357 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:51:32.013577 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:51:32.013578 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.013632 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.013696 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":22,"thread_start_us":82,"threads_started":1}
I20260620 14:51:32.016144 13550 raft_consensus.cc:1261] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Refusing update from remote peer 65e2e4b091ee437d8ccdcb7868521132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:32.016242 13410 raft_consensus.cc:1261] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Refusing update from remote peer 65e2e4b091ee437d8ccdcb7868521132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:32.016390 13671 consensus_queue.cc:1035] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:32.016533 13671 consensus_queue.cc:1035] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:32.019279 13638 mvcc.cc:205] Tried to move back new op lower bound from 7298937208896950272 to 7298937208070631424. Current Snapshot: MvccSnapshot[applied={T|T < 7298937208896950272}]
I20260620 14:51:32.030215 13111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:32.030356 13111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:32.031023 13111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a: Sending DeleteTablet for 3 replicas of tablet af28d5bca1ee48ac94b88e3d5579d53c
I20260620 14:51:32.031273 13530 tablet_service.cc:1514] Processing DeleteTablet for tablet af28d5bca1ee48ac94b88e3d5579d53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:48740
I20260620 14:51:32.031297 13249 tablet_service.cc:1514] Processing DeleteTablet for tablet af28d5bca1ee48ac94b88e3d5579d53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37096
I20260620 14:51:32.031378 13646 tablet_replica.cc:330] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5: stopping tablet replica
I20260620 14:51:32.031378 13648 tablet_replica.cc:330] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132: stopping tablet replica
I20260620 14:51:32.031418 13646 raft_consensus.cc:2229] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:32.031422 13648 raft_consensus.cc:2229] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:32.031455 13646 raft_consensus.cc:2258] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:32.031478 13648 raft_consensus.cc:2258] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:32.031517 13390 tablet_service.cc:1514] Processing DeleteTablet for tablet af28d5bca1ee48ac94b88e3d5579d53c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:59814
I20260620 14:51:32.031607 13647 tablet_replica.cc:330] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b: stopping tablet replica
I20260620 14:51:32.031647 13647 raft_consensus.cc:2229] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:32.031680 13647 raft_consensus.cc:2258] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:32.031822 13648 ts_tablet_manager.cc:1905] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:32.032066 13646 ts_tablet_manager.cc:1905] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:32.033661 13646 ts_tablet_manager.cc:1918] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:32.033740 13646 log.cc:1196] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/af28d5bca1ee48ac94b88e3d5579d53c
I20260620 14:51:32.033952 13646 ts_tablet_manager.cc:1939] T af28d5bca1ee48ac94b88e3d5579d53c P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting consensus metadata
I20260620 14:51:32.034199 13648 ts_tablet_manager.cc:1918] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:32.034255 13648 log.cc:1196] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/wal/wals/af28d5bca1ee48ac94b88e3d5579d53c
I20260620 14:51:32.034407 13100 catalog_manager.cc:4841] TS ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277): tablet af28d5bca1ee48ac94b88e3d5579d53c (table test-table [id=76417b53e576406c907fd7680f42fff2]) successfully deleted
I20260620 14:51:32.034466 13648 ts_tablet_manager.cc:1939] T af28d5bca1ee48ac94b88e3d5579d53c P 65e2e4b091ee437d8ccdcb7868521132: Deleting consensus metadata
I20260620 14:51:32.034708 13647 ts_tablet_manager.cc:1905] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:32.034763 13099 catalog_manager.cc:4841] TS 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129:42167): tablet af28d5bca1ee48ac94b88e3d5579d53c (table test-table [id=76417b53e576406c907fd7680f42fff2]) successfully deleted
I20260620 14:51:32.036232 13647 ts_tablet_manager.cc:1918] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:32.036296 13647 log.cc:1196] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/af28d5bca1ee48ac94b88e3d5579d53c
I20260620 14:51:32.036511 13647 ts_tablet_manager.cc:1939] T af28d5bca1ee48ac94b88e3d5579d53c P f50f447f010c40ff8c65dd826506ac8b: Deleting consensus metadata
I20260620 14:51:32.036837 13097 catalog_manager.cc:4841] TS f50f447f010c40ff8c65dd826506ac8b (127.5.118.130:35843): tablet af28d5bca1ee48ac94b88e3d5579d53c (table test-table [id=76417b53e576406c907fd7680f42fff2]) successfully deleted
Jun 20 14:51:32 dist-test-slave-2m5f krb5kdc[13068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:51:32 dist-test-slave-2m5f krb5kdc[13068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:32 dist-test-slave-2m5f krb5kdc[13068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:51:32.058318 13162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:32.051984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.5.118.190:43231)
0620 14:51:32.052079 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.052082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.052260 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:51:32.052332 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.052333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.052343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.052396 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:32.053126 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.053613 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.054697 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.054884 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.054945 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.058249 (+ 3304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53550: BlockingRecv error: recv got EOF from 127.0.0.1:53550 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 14:51:32 dist-test-slave-2m5f krb5kdc[13068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM
I20260620 14:51:32.066635 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.059077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.5.118.190:43231)
0620 14:51:32.059119 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.059121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.059155 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:51:32.059225 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.059226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.059235 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.059298 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:32.059731 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.060524 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.061251 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.061503 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.061573 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.063794 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:32.063796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:32.063796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:32.063804 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:32.065522 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:32.065867 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:32.065868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:32.065869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:32.065933 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:32.066127 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:32.066128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:32.066128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:32.066264 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:32.066359 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.066449 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.066533 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:51:32.068249 13111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:32.068444 13111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:32.069875 13249 tablet_service.cc:1467] Processing CreateTablet for tablet 9e12bc5587a74f6dafd172227260dc33 (DEFAULT_TABLE table=test-table [id=7a6238a3ad824e2b811334fa14283d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:32.069876 13390 tablet_service.cc:1467] Processing CreateTablet for tablet 9e12bc5587a74f6dafd172227260dc33 (DEFAULT_TABLE table=test-table [id=7a6238a3ad824e2b811334fa14283d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:32.070050 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e12bc5587a74f6dafd172227260dc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:32.070050 13249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e12bc5587a74f6dafd172227260dc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:32.071429 13625 tablet_bootstrap.cc:492] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132: Bootstrap starting.
I20260620 14:51:32.071436 13624 tablet_bootstrap.cc:492] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b: Bootstrap starting.
I20260620 14:51:32.071466 13530 tablet_service.cc:1467] Processing CreateTablet for tablet 9e12bc5587a74f6dafd172227260dc33 (DEFAULT_TABLE table=test-table [id=7a6238a3ad824e2b811334fa14283d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:32.071594 13530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e12bc5587a74f6dafd172227260dc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:32.071760 13625 tablet_bootstrap.cc:654] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:32.071874 13624 tablet_bootstrap.cc:654] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:32.072540 13624 tablet_bootstrap.cc:492] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b: No bootstrap required, opened a new log
I20260620 14:51:32.072592 13624 ts_tablet_manager.cc:1397] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:32.072742 13624 raft_consensus.cc:348] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.072834 13624 raft_consensus.cc:374] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:32.072856 13624 raft_consensus.cc:729] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50f447f010c40ff8c65dd826506ac8b, State: Initialized, Role: FOLLOWER
I20260620 14:51:32.072878 13630 tablet_bootstrap.cc:492] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Bootstrap starting.
I20260620 14:51:32.072889 13624 consensus_queue.cc:260] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.072975 13624 ts_tablet_manager.cc:1428] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:32.073557 13630 tablet_bootstrap.cc:654] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:32.073948 13625 tablet_bootstrap.cc:492] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132: No bootstrap required, opened a new log
I20260620 14:51:32.074003 13625 ts_tablet_manager.cc:1397] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 14:51:32.074110 13625 raft_consensus.cc:348] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.074175 13625 raft_consensus.cc:374] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:32.074190 13630 tablet_bootstrap.cc:492] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: No bootstrap required, opened a new log
I20260620 14:51:32.074196 13625 raft_consensus.cc:729] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e2e4b091ee437d8ccdcb7868521132, State: Initialized, Role: FOLLOWER
I20260620 14:51:32.074223 13630 ts_tablet_manager.cc:1397] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:32.074247 13625 consensus_queue.cc:260] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.074342 13625 ts_tablet_manager.cc:1428] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:32.074395 13630 raft_consensus.cc:348] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.074478 13630 raft_consensus.cc:374] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:32.074502 13630 raft_consensus.cc:729] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba67d8f9c7ad4828b5adf0dc8c8b36c5, State: Initialized, Role: FOLLOWER
I20260620 14:51:32.074550 13630 consensus_queue.cc:260] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.074654 13630 ts_tablet_manager.cc:1428] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:32.299715 13629 raft_consensus.cc:482] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:32.299820 13629 raft_consensus.cc:504] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.300247 13629 leader_election.cc:288] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129:42167), ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277)
I20260620 14:51:32.303273 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.300586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:37825 (local address 127.5.118.129:42167)
0620 14:51:32.300634 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.300636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.300646 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:32.300681 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.300682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.300691 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.300752 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:32.300906 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.301412 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.302579 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.302926 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.302985 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.302994 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.303151 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.303194 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:51:32.303365 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.300405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:44277 (local address 127.5.118.130:35295)
0620 14:51:32.300553 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.300567 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.300628 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.300813 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.300814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.300817 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:32.300910 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.300912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.301548 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.301549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.302375 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.302377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.303088 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.303089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.303177 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.303182 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.303240 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.303278 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260620 14:51:32.303388 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.300500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:35295 (local address 127.5.118.131:44277)
0620 14:51:32.300618 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.300620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.300630 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:32.300697 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.300702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.300711 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.300793 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:32.300984 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.301436 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.302443 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.302940 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.303041 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.303049 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.303263 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.303323 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260620 14:51:32.303633 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.300427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:42167 (local address 127.5.118.130:37825)
0620 14:51:32.300559 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.300572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.300631 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.300767 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.300768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.300771 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:32.300833 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.300836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.301498 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.301499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.302426 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.302430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.303005 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.303006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.303068 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.303072 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.303131 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.303166 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":90,"thread_start_us":59,"threads_started":1}
I20260620 14:51:32.303623 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e12bc5587a74f6dafd172227260dc33" candidate_uuid: "f50f447f010c40ff8c65dd826506ac8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" is_pre_election: true
I20260620 14:51:32.303714 13550 raft_consensus.cc:2454] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f50f447f010c40ff8c65dd826506ac8b in term 0.
I20260620 14:51:32.303934 13344 leader_election.cc:302] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba67d8f9c7ad4828b5adf0dc8c8b36c5, f50f447f010c40ff8c65dd826506ac8b; no voters:
I20260620 14:51:32.303941 13269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e12bc5587a74f6dafd172227260dc33" candidate_uuid: "f50f447f010c40ff8c65dd826506ac8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e2e4b091ee437d8ccdcb7868521132" is_pre_election: true
I20260620 14:51:32.304073 13269 raft_consensus.cc:2454] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f50f447f010c40ff8c65dd826506ac8b in term 0.
I20260620 14:51:32.304116 13629 raft_consensus.cc:2781] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:32.304186 13629 raft_consensus.cc:482] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:32.304220 13629 raft_consensus.cc:3037] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:32.304967 13629 raft_consensus.cc:504] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.305122 13629 leader_election.cc:288] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [CANDIDATE]: Term 1 election: Requested vote from peers 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129:42167), ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277)
I20260620 14:51:32.305270 13269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e12bc5587a74f6dafd172227260dc33" candidate_uuid: "f50f447f010c40ff8c65dd826506ac8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e2e4b091ee437d8ccdcb7868521132"
I20260620 14:51:32.305274 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e12bc5587a74f6dafd172227260dc33" candidate_uuid: "f50f447f010c40ff8c65dd826506ac8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5"
I20260620 14:51:32.305361 13550 raft_consensus.cc:3037] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:32.305364 13269 raft_consensus.cc:3037] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:32.305995 13550 raft_consensus.cc:2454] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f50f447f010c40ff8c65dd826506ac8b in term 1.
I20260620 14:51:32.306042 13269 raft_consensus.cc:2454] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f50f447f010c40ff8c65dd826506ac8b in term 1.
I20260620 14:51:32.306217 13344 leader_election.cc:302] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba67d8f9c7ad4828b5adf0dc8c8b36c5, f50f447f010c40ff8c65dd826506ac8b; no voters:
I20260620 14:51:32.306326 13629 raft_consensus.cc:2781] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:32.306496 13629 raft_consensus.cc:686] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 1 LEADER]: Becoming Leader. State: Replica: f50f447f010c40ff8c65dd826506ac8b, State: Running, Role: LEADER
I20260620 14:51:32.306706 13629 consensus_queue.cc:237] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.307461 13111 catalog_manager.cc:5495] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b reported cstate change: term changed from 0 to 1, leader changed from <none> to f50f447f010c40ff8c65dd826506ac8b (127.5.118.130). New cstate: current_term: 1 leader_uuid: "f50f447f010c40ff8c65dd826506ac8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:32.359028 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.356351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.5.118.130:35843)
0620 14:51:32.356476 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.356477 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.356485 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:32.356525 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.356525 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.356544 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.356642 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:32.356758 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.357322 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.358113 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.358313 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.358366 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.358585 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:32.358695 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:32.358757 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.358806 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.358822 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":74,"threads_started":1}
I20260620 14:51:32.360242 13550 raft_consensus.cc:1261] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Refusing update from remote peer f50f447f010c40ff8c65dd826506ac8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:32.360317 13269 raft_consensus.cc:1261] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Refusing update from remote peer f50f447f010c40ff8c65dd826506ac8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:32.360549 13629 consensus_queue.cc:1035] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:32.360697 13695 consensus_queue.cc:1035] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:32.364245 13111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:32.364364 13111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:32.365041 13111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a: Sending DeleteTablet for 3 replicas of tablet 9e12bc5587a74f6dafd172227260dc33
I20260620 14:51:32.365286 13249 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e12bc5587a74f6dafd172227260dc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37096
I20260620 14:51:32.365393 13648 tablet_replica.cc:330] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132: stopping tablet replica
I20260620 14:51:32.365453 13648 raft_consensus.cc:2229] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:32.365432 13530 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e12bc5587a74f6dafd172227260dc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:48740
I20260620 14:51:32.365501 13648 raft_consensus.cc:2258] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:32.365520 13646 tablet_replica.cc:330] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: stopping tablet replica
I20260620 14:51:32.365562 13646 raft_consensus.cc:2229] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:32.365614 13646 raft_consensus.cc:2258] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:32.365911 13646 ts_tablet_manager.cc:1905] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:32.365933 13390 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e12bc5587a74f6dafd172227260dc33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:59814
I20260620 14:51:32.366024 13647 tablet_replica.cc:330] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b: stopping tablet replica
I20260620 14:51:32.366070 13647 raft_consensus.cc:2229] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:32.366015 13111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:51:32.366160 13647 raft_consensus.cc:2258] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 14:51:32.366163 13111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:32.366472 13647 ts_tablet_manager.cc:1905] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:32.367349 13249 tablet_service.cc:1467] Processing CreateTablet for tablet e718ac46818f46999d033a46f3296f24 (DEFAULT_TABLE table=test-table [id=e01057f0568a4f978c32cddb270d6e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:32.367498 13249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e718ac46818f46999d033a46f3296f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:32.367583 13646 ts_tablet_manager.cc:1918] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:32.367636 13646 log.cc:1196] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/9e12bc5587a74f6dafd172227260dc33
I20260620 14:51:32.367848 13646 ts_tablet_manager.cc:1939] T 9e12bc5587a74f6dafd172227260dc33 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting consensus metadata
I20260620 14:51:32.367990 13647 ts_tablet_manager.cc:1918] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:32.368044 13647 log.cc:1196] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/9e12bc5587a74f6dafd172227260dc33
I20260620 14:51:32.368287 13647 ts_tablet_manager.cc:1939] T 9e12bc5587a74f6dafd172227260dc33 P f50f447f010c40ff8c65dd826506ac8b: Deleting consensus metadata
I20260620 14:51:32.368423 13530 tablet_service.cc:1467] Processing CreateTablet for tablet e718ac46818f46999d033a46f3296f24 (DEFAULT_TABLE table=test-table [id=e01057f0568a4f978c32cddb270d6e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:32.368551 13530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e718ac46818f46999d033a46f3296f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:32.368588 13648 ts_tablet_manager.cc:1905] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:32.369685 13648 ts_tablet_manager.cc:1918] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:32.369745 13648 log.cc:1196] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-0/wal/wals/9e12bc5587a74f6dafd172227260dc33
I20260620 14:51:32.369765 13630 tablet_bootstrap.cc:492] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Bootstrap starting.
I20260620 14:51:32.369953 13648 ts_tablet_manager.cc:1939] T 9e12bc5587a74f6dafd172227260dc33 P 65e2e4b091ee437d8ccdcb7868521132: Deleting consensus metadata
I20260620 14:51:32.370172 13630 tablet_bootstrap.cc:654] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:32.370628 13099 catalog_manager.cc:4841] TS 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129:42167): tablet 9e12bc5587a74f6dafd172227260dc33 (table test-table [id=7a6238a3ad824e2b811334fa14283d6c]) successfully deleted
I20260620 14:51:32.370885 13630 tablet_bootstrap.cc:492] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: No bootstrap required, opened a new log
I20260620 14:51:32.370955 13630 ts_tablet_manager.cc:1397] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:51:32.371100 13630 raft_consensus.cc:348] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.371191 13630 raft_consensus.cc:374] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:32.371217 13630 raft_consensus.cc:729] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba67d8f9c7ad4828b5adf0dc8c8b36c5, State: Initialized, Role: FOLLOWER
I20260620 14:51:32.371260 13630 consensus_queue.cc:260] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.371342 13630 ts_tablet_manager.cc:1428] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:32.371721 13625 tablet_bootstrap.cc:492] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132: Bootstrap starting.
I20260620 14:51:32.371804 13390 tablet_service.cc:1467] Processing CreateTablet for tablet e718ac46818f46999d033a46f3296f24 (DEFAULT_TABLE table=test-table [id=e01057f0568a4f978c32cddb270d6e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:32.371932 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e718ac46818f46999d033a46f3296f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:32.372000 13625 tablet_bootstrap.cc:654] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:32.372525 13625 tablet_bootstrap.cc:492] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132: No bootstrap required, opened a new log
I20260620 14:51:32.372578 13625 ts_tablet_manager.cc:1397] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 14:51:32.372690 13625 raft_consensus.cc:348] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.372781 13625 raft_consensus.cc:374] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:32.372809 13625 raft_consensus.cc:729] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e2e4b091ee437d8ccdcb7868521132, State: Initialized, Role: FOLLOWER
I20260620 14:51:32.372851 13625 consensus_queue.cc:260] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.372952 13625 ts_tablet_manager.cc:1428] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:32.373524 13624 tablet_bootstrap.cc:492] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b: Bootstrap starting.
I20260620 14:51:32.373924 13624 tablet_bootstrap.cc:654] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:32.374567 13097 catalog_manager.cc:4841] TS f50f447f010c40ff8c65dd826506ac8b (127.5.118.130:35843): tablet 9e12bc5587a74f6dafd172227260dc33 (table test-table [id=7a6238a3ad824e2b811334fa14283d6c]) successfully deleted
I20260620 14:51:32.374624 13100 catalog_manager.cc:4841] TS ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277): tablet 9e12bc5587a74f6dafd172227260dc33 (table test-table [id=7a6238a3ad824e2b811334fa14283d6c]) successfully deleted
I20260620 14:51:32.374852 13624 tablet_bootstrap.cc:492] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b: No bootstrap required, opened a new log
I20260620 14:51:32.374903 13624 ts_tablet_manager.cc:1397] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:32.375083 13624 raft_consensus.cc:348] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.375156 13624 raft_consensus.cc:374] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:32.375178 13624 raft_consensus.cc:729] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50f447f010c40ff8c65dd826506ac8b, State: Initialized, Role: FOLLOWER
I20260620 14:51:32.375219 13624 consensus_queue.cc:260] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.375311 13624 ts_tablet_manager.cc:1428] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:32.512041 13629 raft_consensus.cc:482] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:32.512163 13629 raft_consensus.cc:504] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.512395 13629 leader_election.cc:288] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129:42167), ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277)
I20260620 14:51:32.512609 13269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e718ac46818f46999d033a46f3296f24" candidate_uuid: "f50f447f010c40ff8c65dd826506ac8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e2e4b091ee437d8ccdcb7868521132" is_pre_election: true
I20260620 14:51:32.512737 13269 raft_consensus.cc:2454] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f50f447f010c40ff8c65dd826506ac8b in term 0.
I20260620 14:51:32.512701 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e718ac46818f46999d033a46f3296f24" candidate_uuid: "f50f447f010c40ff8c65dd826506ac8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" is_pre_election: true
I20260620 14:51:32.512779 13550 raft_consensus.cc:2454] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f50f447f010c40ff8c65dd826506ac8b in term 0.
I20260620 14:51:32.512894 13343 leader_election.cc:302] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65e2e4b091ee437d8ccdcb7868521132, f50f447f010c40ff8c65dd826506ac8b; no voters:
I20260620 14:51:32.513042 13629 raft_consensus.cc:2781] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:32.513080 13629 raft_consensus.cc:482] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:32.513104 13629 raft_consensus.cc:3037] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:32.513821 13629 raft_consensus.cc:504] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.513985 13629 leader_election.cc:288] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [CANDIDATE]: Term 1 election: Requested vote from peers 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129:42167), ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277)
I20260620 14:51:32.514130 13269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e718ac46818f46999d033a46f3296f24" candidate_uuid: "f50f447f010c40ff8c65dd826506ac8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65e2e4b091ee437d8ccdcb7868521132"
I20260620 14:51:32.514171 13550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e718ac46818f46999d033a46f3296f24" candidate_uuid: "f50f447f010c40ff8c65dd826506ac8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5"
I20260620 14:51:32.514216 13269 raft_consensus.cc:3037] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:32.514241 13550 raft_consensus.cc:3037] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:32.514966 13269 raft_consensus.cc:2454] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f50f447f010c40ff8c65dd826506ac8b in term 1.
I20260620 14:51:32.514966 13550 raft_consensus.cc:2454] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f50f447f010c40ff8c65dd826506ac8b in term 1.
I20260620 14:51:32.515161 13344 leader_election.cc:302] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba67d8f9c7ad4828b5adf0dc8c8b36c5, f50f447f010c40ff8c65dd826506ac8b; no voters:
I20260620 14:51:32.515271 13629 raft_consensus.cc:2781] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:32.515372 13629 raft_consensus.cc:686] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 1 LEADER]: Becoming Leader. State: Replica: f50f447f010c40ff8c65dd826506ac8b, State: Running, Role: LEADER
I20260620 14:51:32.515426 13629 consensus_queue.cc:237] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } }
I20260620 14:51:32.516050 13111 catalog_manager.cc:5495] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b reported cstate change: term changed from 0 to 1, leader changed from <none> to f50f447f010c40ff8c65dd826506ac8b (127.5.118.130). New cstate: current_term: 1 leader_uuid: "f50f447f010c40ff8c65dd826506ac8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f50f447f010c40ff8c65dd826506ac8b" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:32.530227 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.527721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:42167 (local address 127.5.118.130:48995)
0620 14:51:32.527869 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.527881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.527941 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.528072 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.528072 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.528074 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:51:32.528119 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.528120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.528865 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.528866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.529463 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.529464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.529840 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.529842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.529899 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.529915 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:51:32.530093 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:51:32.530094 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.530126 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.530166 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260620 14:51:32.530220 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.527848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:48995 (local address 127.5.118.129:42167)
0620 14:51:32.527893 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.527896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.527931 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:51:32.527989 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.527990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.527999 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.528064 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:32.528157 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.528750 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.529564 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.529763 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.529825 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.529994 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:32.530024 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:32.530082 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.530138 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.530156 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:51:32.532557 13269 raft_consensus.cc:1261] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Refusing update from remote peer f50f447f010c40ff8c65dd826506ac8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:32.532557 13550 raft_consensus.cc:1261] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Refusing update from remote peer f50f447f010c40ff8c65dd826506ac8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:32.532878 13695 consensus_queue.cc:1035] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba67d8f9c7ad4828b5adf0dc8c8b36c5" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:32.533001 13629 consensus_queue.cc:1035] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e2e4b091ee437d8ccdcb7868521132" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 42167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:32.540886 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.537998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:43363 (local address 127.5.118.130:35843)
0620 14:51:32.538043 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:32.538045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:32.538161 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:51:32.538234 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:32.538235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:32.538247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:32.538308 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:32.538455 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.538995 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.540044 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.540374 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.540434 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.540651 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:32.540680 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:32.540722 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:32.540769 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 14:51:32.540791 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:51:32.540903 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.537941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:35843 (local address 127.5.118.129:43363)
0620 14:51:32.538099 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.538113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:32.538178 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:32.538311 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:32.538312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:32.538314 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:51:32.538369 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.538370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.539103 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.539104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.539955 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:32.539957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:32.540522 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:32.540523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:32.540573 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:32.540578 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:51:32.540731 (+ 153us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:51:32.540732 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 14:51:32.540768 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 14:51:32.540806 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260620 14:51:32.550185 13112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:32.550290 13112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:32.551017 13112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e749aa343f00450e89ab1c4c6c7fa87a: Sending DeleteTablet for 3 replicas of tablet e718ac46818f46999d033a46f3296f24
I20260620 14:51:32.551249 13249 tablet_service.cc:1514] Processing DeleteTablet for tablet e718ac46818f46999d033a46f3296f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37096
I20260620 14:51:32.551312 13390 tablet_service.cc:1514] Processing DeleteTablet for tablet e718ac46818f46999d033a46f3296f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:59814
I20260620 14:51:32.551376 13648 tablet_replica.cc:330] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132: stopping tablet replica
I20260620 14:51:32.551388 13530 tablet_service.cc:1514] Processing DeleteTablet for tablet e718ac46818f46999d033a46f3296f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:32 UTC) from {username='oryx', principal='oryx/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:48740
I20260620 14:51:32.551429 13648 raft_consensus.cc:2229] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:32.551489 13648 raft_consensus.cc:2258] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:32.551518 13647 tablet_replica.cc:330] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b: stopping tablet replica
I20260620 14:51:32.551514 13646 tablet_replica.cc:330] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: stopping tablet replica
I20260620 14:51:32.551573 13646 raft_consensus.cc:2229] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:32.551575 13647 raft_consensus.cc:2229] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:32.551620 13646 raft_consensus.cc:2258] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:32.551645 13647 raft_consensus.cc:2258] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:32.551911 13646 ts_tablet_manager.cc:1905] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:32.551914 13648 ts_tablet_manager.cc:1905] T e718ac46818f46999d033a46f3296f24 P 65e2e4b091ee437d8ccdcb7868521132: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:32.552044 13647 ts_tablet_manager.cc:1905] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:32.552210 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 13185
I20260620 14:51:32.553839 13646 ts_tablet_manager.cc:1918] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:32.553819 13647 ts_tablet_manager.cc:1918] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:51:32.553882 13646 log.cc:1196] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/e718ac46818f46999d033a46f3296f24
I20260620 14:51:32.553889 13647 log.cc:1196] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/e718ac46818f46999d033a46f3296f24
I20260620 14:51:32.554054 13646 ts_tablet_manager.cc:1939] T e718ac46818f46999d033a46f3296f24 P ba67d8f9c7ad4828b5adf0dc8c8b36c5: Deleting consensus metadata
I20260620 14:51:32.554131 13647 ts_tablet_manager.cc:1939] T e718ac46818f46999d033a46f3296f24 P f50f447f010c40ff8c65dd826506ac8b: Deleting consensus metadata
I20260620 14:51:32.554364 13100 catalog_manager.cc:4841] TS ba67d8f9c7ad4828b5adf0dc8c8b36c5 (127.5.118.131:44277): tablet e718ac46818f46999d033a46f3296f24 (table test-table [id=e01057f0568a4f978c32cddb270d6e3e]) successfully deleted
I20260620 14:51:32.554423 13097 catalog_manager.cc:4841] TS f50f447f010c40ff8c65dd826506ac8b (127.5.118.130:35843): tablet e718ac46818f46999d033a46f3296f24 (table test-table [id=e01057f0568a4f978c32cddb270d6e3e]) successfully deleted
W20260620 14:51:32.560457 13099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:42167 (error 108)
I20260620 14:51:32.560736 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 13325
I20260620 14:51:32.562650 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:32.560920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:42167 (local address 127.0.0.1:37136)
0620 14:51:32.562515 (+ 1595us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:32.562571 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:42167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1577,"thread_start_us":46,"threads_started":1}
W20260620 14:51:32.562780 13099 catalog_manager.cc:4584] TS 65e2e4b091ee437d8ccdcb7868521132 (127.5.118.129:42167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e718ac46818f46999d033a46f3296f24: Network error: Client connection negotiation failed: client connection to 127.5.118.129:42167: connect: Connection refused (error 111)
I20260620 14:51:32.567294 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 13465
I20260620 14:51:32.572533 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 13081
2026-06-20T14:51:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2791 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:32 dist-test-slave-2m5f krb5kdc[13719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:32 dist-test-slave-2m5f krb5kdc[13719](info): set up 2 sockets
Jun 20 14:51:32 dist-test-slave-2m5f krb5kdc[13719](info): commencing operation
krb5kdc: starting...
W20260620 14:51:33.704907 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:33 dist-test-slave-2m5f krb5kdc[13719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:51:33.827171 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:44069
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:37331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:44069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:33.910036 13732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:33.910312 13732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:33.910354 13732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:33.911564 13732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:33.911633 13732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:33.911656 13732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:33.911679 13732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:33.911708 13732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:33.913273 13732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:37331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:44069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:44069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:33.913566 13732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:33.913800 13732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:33.918814 13738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:33.918998 13732 server_base.cc:1034] running on GCE node
W20260620 14:51:33.919155 13740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:33.919190 13737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:33.919536 13732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:33.919800 13732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:33.920934 13732 hybrid_clock.cc:648] HybridClock initialized: now 1781967093920922 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:51:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1350 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:33 dist-test-slave-2m5f krb5kdc[13745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:33 dist-test-slave-2m5f krb5kdc[13745](info): set up 2 sockets
Jun 20 14:51:33 dist-test-slave-2m5f krb5kdc[13745](info): commencing operation
krb5kdc: starting...
W20260620 14:51:34.988932 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:35 dist-test-slave-2m5f krb5kdc[13745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:51:35.107638 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42731
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:42731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:35.190640 13758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:35.190903 13758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:35.190943 13758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:35.192272 13758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:35.192333 13758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:35.192358 13758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:35.192384 13758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:35.192414 13758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:35.193949 13758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:42731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:35.194243 13758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:35.194489 13758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:35.199676 13766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:35.199694 13764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:35.199681 13763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:35.199921 13758 server_base.cc:1034] running on GCE node
I20260620 14:51:35.200230 13758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:35.200592 13758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:35.201805 13758 hybrid_clock.cc:648] HybridClock initialized: now 1781967095201778 us; error 37 us; skew 500 ppm
Jun 20 14:51:35 dist-test-slave-2m5f krb5kdc[13745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:35.209460 13758 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:51:35.209959 13758 webserver.cc:460] Webserver started at http://127.5.118.190:37507/ using document root <none> and password file <none>
I20260620 14:51:35.210199 13758 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:35.210271 13758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:35.210449 13758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:35.211570 13758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "c3f4deed2052405b8e29452ab5c28e75"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "56533b544e469936da26082c908dab2f"
server_key_iv: "e07f75426d0d34c8064a9c8874986587"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.211943 13758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "c3f4deed2052405b8e29452ab5c28e75"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "56533b544e469936da26082c908dab2f"
server_key_iv: "e07f75426d0d34c8064a9c8874986587"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.213477 13758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:35.214339 13773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:35.214552 13758 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:35.214660 13758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "c3f4deed2052405b8e29452ab5c28e75"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "56533b544e469936da26082c908dab2f"
server_key_iv: "e07f75426d0d34c8064a9c8874986587"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.214748 13758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:35.226938 13758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:35.229195 13758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:35.229377 13758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:35.233484 13758 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:42731
I20260620 14:51:35.233517 13825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:42731 every 8 connection(s)
I20260620 14:51:35.233881 13758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:35.234756 13826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:35.237165 13826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75: Bootstrap starting.
I20260620 14:51:35.237778 13826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:35.238083 13826 log.cc:824] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:35.238893 13826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75: No bootstrap required, opened a new log
I20260620 14:51:35.240621 13826 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } }
I20260620 14:51:35.240761 13826 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:35.240783 13826 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f4deed2052405b8e29452ab5c28e75, State: Initialized, Role: FOLLOWER
I20260620 14:51:35.240959 13826 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } }
I20260620 14:51:35.241031 13826 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:35.241074 13826 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:35.241125 13826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:35.241691 13826 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } }
I20260620 14:51:35.241779 13826 leader_election.cc:302] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f4deed2052405b8e29452ab5c28e75; no voters:
I20260620 14:51:35.241956 13826 leader_election.cc:288] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:35.242057 13829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:35.242233 13829 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 1 LEADER]: Becoming Leader. State: Replica: c3f4deed2052405b8e29452ab5c28e75, State: Running, Role: LEADER
I20260620 14:51:35.242272 13826 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:35.242331 13829 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } }
I20260620 14:51:35.242749 13830 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3f4deed2052405b8e29452ab5c28e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } } }
I20260620 14:51:35.242753 13831 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f4deed2052405b8e29452ab5c28e75. Latest consensus state: current_term: 1 leader_uuid: "c3f4deed2052405b8e29452ab5c28e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } } }
I20260620 14:51:35.242885 13830 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:35.242902 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 13758
I20260620 14:51:35.242909 13831 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:35.243010 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:35.243141 13833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:35.243214 5594 external_mini_cluster.cc:1428] Setting key 7c79117e646cb31cf00c2206baa78105
I20260620 14:51:35.245327 13833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:35.247148 13833 catalog_manager.cc:1347] Generated new cluster ID: dc1bd2d4c3ea4dc7b7970d67820a926f
I20260620 14:51:35.247203 13833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:51:35 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:35.255261 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:35.244084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.5.118.190:42731)
0620 14:51:35.244352 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:35.244434 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:35.244464 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:51:35.244596 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:35.244605 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:35.244680 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:35.244859 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:35.245777 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.247156 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.248056 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.248653 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.248795 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:35.251453 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:35.251458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:35.251459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:35.251474 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:35.253479 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:35.253898 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:35.253901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:35.253902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:35.254019 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:35.254262 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:35.254263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:35.254263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:35.254551 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:35.254729 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:35.254852 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:51:35.255019 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":158,"threads_started":1}
I20260620 14:51:35.260099 13833 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:35.260658 13833 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:35.265398 13833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75: Generated new TSK 0
I20260620 14:51:35.265657 13833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:51:35.294548 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42731
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:35.403095 13852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:35.403299 13852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:35.403323 13852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:35.404582 13852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:35.404623 13852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:35.404651 13852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:35.406160 13852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:35.406406 13852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:35.406701 13852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:35.411788 13857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:35.411788 13860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:35.411765 13858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:35.412040 13852 server_base.cc:1034] running on GCE node
I20260620 14:51:35.412297 13852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:35.412554 13852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:35.413719 13852 hybrid_clock.cc:648] HybridClock initialized: now 1781967095413713 us; error 36 us; skew 500 ppm
Jun 20 14:51:35 dist-test-slave-2m5f krb5kdc[13745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:35.416340 13852 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:51:35.416788 13852 webserver.cc:460] Webserver started at http://127.5.118.129:40249/ using document root <none> and password file <none>
I20260620 14:51:35.416972 13852 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:35.417013 13852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:35.417150 13852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:35.418649 13852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "88db8278aefb4707a46f1836c6ba1f46"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "aeddc66ca06cbe380b9a3e698ba8fc6a"
server_key_iv: "cd34b1f0e8769148992794f1de28d451"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.418967 13852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "88db8278aefb4707a46f1836c6ba1f46"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "aeddc66ca06cbe380b9a3e698ba8fc6a"
server_key_iv: "cd34b1f0e8769148992794f1de28d451"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.420285 13852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:35.421032 13867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:35.421239 13852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:35.421334 13852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "88db8278aefb4707a46f1836c6ba1f46"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "aeddc66ca06cbe380b9a3e698ba8fc6a"
server_key_iv: "cd34b1f0e8769148992794f1de28d451"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.421406 13852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:35.433925 13852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:35.435871 13852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:35.436038 13852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:35.436319 13852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:35.436652 13852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:35.436738 13852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:35.436781 13852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:35.436810 13852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:35.442539 13852 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:35405
I20260620 14:51:35.442610 13980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:35405 every 8 connection(s)
I20260620 14:51:35.442978 13852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:51:35 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:35.450547 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 13852
I20260620 14:51:35.450660 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:35.450899 5594 external_mini_cluster.cc:1428] Setting key 84f7ec468a46941221b01443a182d640
I20260620 14:51:35.452873 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:35.443701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:60847 (local address 127.5.118.190:42731)
0620 14:51:35.443771 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:35.443775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:35.444003 (+ 228us) server_negotiation.cc:407] Connection header received
0620 14:51:35.444608 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:35.444610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:35.444620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:35.444674 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:35.445666 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.446234 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.447223 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.447416 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.447504 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:35.449779 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:35.449780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:35.449781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:35.449787 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:35.451288 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:35.451833 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:35.451835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:35.451835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:35.451888 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:35.452216 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:35.452224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:35.452225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:35.452344 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:35.452399 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:35.452740 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 14:51:35.452810 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:51:35.453236 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:35.443694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:60847)
0620 14:51:35.443872 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:35.443910 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:35.444514 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:35.444728 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:35.444732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:35.445377 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:35.445586 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:35.445591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.446315 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.446316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:35.447150 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:35.447153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.447517 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.447517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:35.447621 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:35.448225 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:35.448234 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:35.449690 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:35.451400 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:35.451406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:35.451408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:35.451761 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:35.451959 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:35.451960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:35.451960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:35.452110 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:35.452410 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:35.452416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:35.452600 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 14:51:35.452717 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:51:35.452895 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260620 14:51:35.453733 13981 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42731
I20260620 14:51:35.453877 13981 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:35.454087 13981 heartbeater.cc:511] Master 127.5.118.190:42731 requested a full tablet report, sending...
I20260620 14:51:35.454654 13790 ts_manager.cc:194] Registered new tserver with Master: 88db8278aefb4707a46f1836c6ba1f46 (127.5.118.129:35405)
I20260620 14:51:35.455602 13790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:60847
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:51:35.478511 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42731
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:35.560434 13986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:35.560653 13986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:35.560678 13986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:35.561878 13986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:35.561923 13986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:35.561951 13986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:35.563409 13986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:35.563669 13986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:35.563901 13986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:51:35.569272 13986 server_base.cc:1034] running on GCE node
W20260620 14:51:35.569247 13991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:35.569217 13992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:35.569239 13994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:35.569659 13986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:35.569972 13986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:35.571151 13986 hybrid_clock.cc:648] HybridClock initialized: now 1781967095571136 us; error 30 us; skew 500 ppm
Jun 20 14:51:35 dist-test-slave-2m5f krb5kdc[13745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:35.573799 13986 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:51:35.574223 13986 webserver.cc:460] Webserver started at http://127.5.118.130:41837/ using document root <none> and password file <none>
I20260620 14:51:35.574429 13986 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:35.574471 13986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:35.574563 13986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:35.575567 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "895edd90025a403a83a58eaf5f649112"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "13a99e99ee88857afefddd7fff98c9d5"
server_key_iv: "b913a79ebe96946e367ae1552dfa411b"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.575857 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "895edd90025a403a83a58eaf5f649112"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "13a99e99ee88857afefddd7fff98c9d5"
server_key_iv: "b913a79ebe96946e367ae1552dfa411b"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.577244 13986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:51:35.577993 14001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:35.578194 13986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:35.578289 13986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "895edd90025a403a83a58eaf5f649112"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "13a99e99ee88857afefddd7fff98c9d5"
server_key_iv: "b913a79ebe96946e367ae1552dfa411b"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.578343 13986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:35.598655 13986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:35.600932 13986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:35.601068 13986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:35.601258 13986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:35.601590 13986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:35.601625 13986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:35.601649 13986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:35.601701 13986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:35.607674 13986 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:41799
I20260620 14:51:35.607722 14114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:41799 every 8 connection(s)
I20260620 14:51:35.608070 13986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:35.613808 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 13986
I20260620 14:51:35.613955 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:35.614159 5594 external_mini_cluster.cc:1428] Setting key 3983b4b3c4a2af50d4d7f755d5b2e3ff
Jun 20 14:51:35 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:35.618795 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:35.608911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:59217 (local address 127.5.118.190:42731)
0620 14:51:35.608962 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:35.608964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:35.609197 (+ 233us) server_negotiation.cc:407] Connection header received
0620 14:51:35.609755 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:35.609756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:35.609769 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:35.609817 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:35.610915 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.611434 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.612550 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.612756 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.612844 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:35.615374 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:35.615375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:35.615376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:35.615383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:35.617119 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:35.617639 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:35.617641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:35.617641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:35.617710 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:35.618028 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:35.618029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:35.618030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:35.618201 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:35.618282 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:35.618592 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 14:51:35.618683 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:51:35.619036 14117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:35.608892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:59217)
0620 14:51:35.609070 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:35.609107 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:35.609631 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:35.609893 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:35.609898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:35.610583 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:35.610807 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:35.610813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.611543 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.611544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:35.612459 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:35.612462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.612858 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.612858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:35.612987 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:35.613579 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:35.613587 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:35.615277 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:35.617217 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:35.617225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:35.617228 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:35.617489 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:35.617782 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:35.617783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:35.617784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:35.617920 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:35.618272 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:35.618279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:35.618442 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 14:51:35.618568 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:51:35.618757 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260620 14:51:35.619482 14115 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42731
I20260620 14:51:35.619591 14115 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:35.619837 14115 heartbeater.cc:511] Master 127.5.118.190:42731 requested a full tablet report, sending...
I20260620 14:51:35.620299 13790 ts_manager.cc:194] Registered new tserver with Master: 895edd90025a403a83a58eaf5f649112 (127.5.118.130:41799)
I20260620 14:51:35.620728 13790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:59217
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:51:35.639032 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42731
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:35.723882 14120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:35.724094 14120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:35.724118 14120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:35.725449 14120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:35.725503 14120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:35.725533 14120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:35.726972 14120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:35.727172 14120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:35.727367 14120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:35.732746 14128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:35.732758 14126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:35.732746 14125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:35.733067 14120 server_base.cc:1034] running on GCE node
I20260620 14:51:35.733275 14120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:35.733534 14120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:35.734686 14120 hybrid_clock.cc:648] HybridClock initialized: now 1781967095734673 us; error 36 us; skew 500 ppm
Jun 20 14:51:35 dist-test-slave-2m5f krb5kdc[13745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:35.737421 14120 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:51:35.737912 14120 webserver.cc:460] Webserver started at http://127.5.118.131:44879/ using document root <none> and password file <none>
I20260620 14:51:35.738180 14120 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:35.738227 14120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:35.738391 14120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:35.739589 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "2a99fccc1c6f45069f1a635aaf44eb9e"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "11e502f11e3bea3d2f7d7e25af615713"
server_key_iv: "abfd756205667d4b5085c32541d0fbe9"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.739956 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "2a99fccc1c6f45069f1a635aaf44eb9e"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "11e502f11e3bea3d2f7d7e25af615713"
server_key_iv: "abfd756205667d4b5085c32541d0fbe9"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.741439 14120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:35.742309 14135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:35.742551 14120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:35.742659 14120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "2a99fccc1c6f45069f1a635aaf44eb9e"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "11e502f11e3bea3d2f7d7e25af615713"
server_key_iv: "abfd756205667d4b5085c32541d0fbe9"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.742745 14120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:35.755417 14120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:35.757894 14120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:35.758067 14120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:35.758307 14120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:35.758618 14120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:35.758671 14120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:35.758715 14120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:35.758754 14120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:35.764667 14120 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:36931
I20260620 14:51:35.764712 14248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:36931 every 8 connection(s)
I20260620 14:51:35.765061 14120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:51:35 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:35.774442 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 14120
I20260620 14:51:35.774562 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:35.774753 5594 external_mini_cluster.cc:1428] Setting key 3bcf28db3411c0170557540f854b7d39
I20260620 14:51:35.775065 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:35.765852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:42819 (local address 127.5.118.190:42731)
0620 14:51:35.765917 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:35.765919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:35.766220 (+ 301us) server_negotiation.cc:407] Connection header received
0620 14:51:35.766757 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:35.766758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:35.766771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:35.766817 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:35.767823 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.768412 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.769457 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.769652 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.769734 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:35.772107 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:35.772109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:35.772109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:35.772117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:35.773495 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:35.773887 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:35.773889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:35.773889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:35.773952 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:35.774173 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:35.774174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:35.774174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:35.774405 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:35.774488 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:35.774878 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 14:51:35.774971 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:51:35.775311 14251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:35.765888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:42819)
0620 14:51:35.766091 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:35.766128 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:35.766642 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:35.766890 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:35.766894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:35.767514 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:35.767719 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:35.767724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.768504 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.768505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:35.769339 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:35.769341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:35.769744 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:35.769746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:35.769874 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:35.770440 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:35.770453 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:35.772002 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:35.773578 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:35.773585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:35.773587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:35.773811 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:35.774013 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:35.774013 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:35.774013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:35.774115 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:35.774529 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:35.774535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:35.774712 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 14:51:35.774830 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:51:35.775040 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":131,"threads_started":1}
I20260620 14:51:35.775851 14249 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42731
I20260620 14:51:35.776012 14249 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:35.776270 14249 heartbeater.cc:511] Master 127.5.118.190:42731 requested a full tablet report, sending...
I20260620 14:51:35.776667 13790 ts_manager.cc:194] Registered new tserver with Master: 2a99fccc1c6f45069f1a635aaf44eb9e (127.5.118.131:36931)
I20260620 14:51:35.777046 13790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:42819
I20260620 14:51:35.786819 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab.
Entry for principal oryx/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab.
WARNING: no policy specified for oryx/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab.
Entry for principal oryx/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab.
WARNING: no policy specified for oryx/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab.
Entry for principal oryx/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab.
I20260620 14:51:35.859378 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 13852
I20260620 14:51:35.863996 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 13986
I20260620 14:51:35.868083 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 14120
I20260620 14:51:35.872561 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 13758
I20260620 14:51:35.878505 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42731
--webserver_interface=127.5.118.190
--webserver_port=37507
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:42731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:35.971753 14258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:35.971998 14258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:35.972033 14258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:35.973423 14258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:35.973475 14258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:35.973490 14258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:35.973503 14258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:35.973516 14258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:35.975023 14258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:42731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=37507
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:35.975231 14258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:35.975418 14258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:35.980474 14264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:35.980454 14266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:35.980437 14258 server_base.cc:1034] running on GCE node
W20260620 14:51:35.980522 14263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:35.980926 14258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:35.981289 14258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:35.982540 14258 hybrid_clock.cc:648] HybridClock initialized: now 1781967095982486 us; error 67 us; skew 500 ppm
Jun 20 14:51:35 dist-test-slave-2m5f krb5kdc[13745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967095, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:35.985132 14258 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:51:35.985610 14258 webserver.cc:460] Webserver started at http://127.5.118.190:37507/ using document root <none> and password file <none>
I20260620 14:51:35.985850 14258 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:35.985935 14258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:35.987450 14258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:35.988294 14273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:35.988539 14258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:35.988642 14258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "c3f4deed2052405b8e29452ab5c28e75"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "56533b544e469936da26082c908dab2f"
server_key_iv: "e07f75426d0d34c8064a9c8874986587"
server_key_version: "encryptionkey@0"
I20260620 14:51:35.989027 14258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:36.001983 14258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:36.004022 14258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:36.004273 14258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:36.008464 14258 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:42731
I20260620 14:51:36.008507 14325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:42731 every 8 connection(s)
I20260620 14:51:36.008879 14258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:36.010423 14326 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:51:36.011099 14326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75: Bootstrap starting.
I20260620 14:51:36.013562 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 14258
I20260620 14:51:36.013880 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:35405
--local_ip_for_outbound_sockets=127.5.118.129
--tserver_master_addrs=127.5.118.190:42731
--webserver_port=40249
--webserver_interface=127.5.118.129
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.118.129
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:51:36.020241 14326 log.cc:824] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:36.022064 14326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:51:36.022428 14326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75: Bootstrap complete.
I20260620 14:51:36.024873 14326 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } }
I20260620 14:51:36.025147 14326 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f4deed2052405b8e29452ab5c28e75, State: Initialized, Role: FOLLOWER
I20260620 14:51:36.025346 14326 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } }
I20260620 14:51:36.025427 14326 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:36.025481 14326 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:36.025533 14326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:51:36.026571 14326 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } }
I20260620 14:51:36.026695 14326 leader_election.cc:302] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f4deed2052405b8e29452ab5c28e75; no voters:
I20260620 14:51:36.026908 14326 leader_election.cc:288] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:51:36.026945 14330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:51:36.027061 14330 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [term 2 LEADER]: Becoming Leader. State: Replica: c3f4deed2052405b8e29452ab5c28e75, State: Running, Role: LEADER
I20260620 14:51:36.027153 14330 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } }
I20260620 14:51:36.027200 14326 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:36.027426 14330 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f4deed2052405b8e29452ab5c28e75. Latest consensus state: current_term: 2 leader_uuid: "c3f4deed2052405b8e29452ab5c28e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } } }
I20260620 14:51:36.027504 14330 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:36.027431 14331 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c3f4deed2052405b8e29452ab5c28e75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f4deed2052405b8e29452ab5c28e75" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42731 } } }
I20260620 14:51:36.027760 14331 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:36.027701 14334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:36.028517 14334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:36.028676 14334 catalog_manager.cc:1259] Loaded cluster ID: dc1bd2d4c3ea4dc7b7970d67820a926f
I20260620 14:51:36.028838 14334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:36.030113 14334 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:36.030303 14334 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c3f4deed2052405b8e29452ab5c28e75: Loaded TSK: 0
I20260620 14:51:36.030517 14334 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:51:36.103523 14328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:36.103742 14328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:36.103770 14328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:36.105842 14328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:36.105990 14328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:36.106045 14328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:36.108757 14328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.129.keytab
--principal=oryx/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:35405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=40249
--tserver_master_addrs=127.5.118.190:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:36.109104 14328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:36.109405 14328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:36.115868 14352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:36.116007 14328 server_base.cc:1034] running on GCE node
W20260620 14:51:36.115875 14353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:36.116148 14355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:36.116420 14328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:36.116721 14328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:36.117879 14328 hybrid_clock.cc:648] HybridClock initialized: now 1781967096117871 us; error 29 us; skew 500 ppm
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967096, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:36.120653 14328 init.cc:377] Logged in from keytab as oryx/127.5.118.129@KRBTEST.COM (short username oryx)
I20260620 14:51:36.121230 14328 webserver.cc:460] Webserver started at http://127.5.118.129:40249/ using document root <none> and password file <none>
I20260620 14:51:36.121517 14328 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:36.121603 14328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:36.123456 14328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:36.124118 14362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:36.124356 14328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:36.124468 14328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "88db8278aefb4707a46f1836c6ba1f46"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "aeddc66ca06cbe380b9a3e698ba8fc6a"
server_key_iv: "cd34b1f0e8769148992794f1de28d451"
server_key_version: "encryptionkey@0"
I20260620 14:51:36.124826 14328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:36.138811 14328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:36.141304 14328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:36.141451 14328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:36.141717 14328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:36.142047 14328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:36.142107 14328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:36.142150 14328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:36.142179 14328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:36.148692 14328 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:35405
I20260620 14:51:36.148743 14475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:35405 every 8 connection(s)
I20260620 14:51:36.149137 14328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:36.149708 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 14328
I20260620 14:51:36.150036 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:41799
--local_ip_for_outbound_sockets=127.5.118.130
--tserver_master_addrs=127.5.118.190:42731
--webserver_port=41837
--webserver_interface=127.5.118.130
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.118.130
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:51:36.161814 14479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:36.150155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:42867 (local address 127.5.118.190:42731)
0620 14:51:36.150374 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:36.150384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:36.150632 (+ 248us) server_negotiation.cc:407] Connection header received
0620 14:51:36.151451 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:36.151459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:36.151517 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:36.151637 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:36.153200 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.154643 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.156274 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.157143 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.157247 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.161564 (+ 4317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:42867: BlockingRecv error: recv got EOF from 127.5.118.129:42867 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260620 14:51:36.162220 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:36.150085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:42867)
0620 14:51:36.150472 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:36.150523 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:36.151215 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:36.151859 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:36.151863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:36.152737 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:36.153047 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.153055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.154803 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.154804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.156122 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.156125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.157403 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.157404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.157577 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.158349 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:36.158361 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:36.162029 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":83,"threads_started":1}
W20260620 14:51:36.162460 14476 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:42731 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.118.190:42731: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:36.168968 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:36.162683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:57999)
0620 14:51:36.162869 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:36.162882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:36.162955 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:36.163154 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:36.163155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:36.163337 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:36.163401 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.163403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.164436 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.164438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.165349 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.165352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.165895 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.165897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.165974 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.166573 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:36.166579 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:36.168863 (+ 2284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":96,"threads_started":1}
W20260620 14:51:36.169656 14479 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:36.162899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:57999 (local address 127.5.118.190:42731)
0620 14:51:36.162937 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:36.162939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:36.162949 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:36.163058 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:36.163059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:36.163073 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:36.163139 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:36.163491 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.164348 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.165461 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.165786 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.165864 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.168998 (+ 3134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:57999: BlockingRecv error: recv got EOF from 127.5.118.129:57999 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:36.175731 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:36.169413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:42379)
0620 14:51:36.169935 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:36.169948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:36.170007 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:36.170179 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:36.170180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:36.170407 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:36.170464 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.170468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.171408 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.171409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.172408 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.172412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.172927 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.172928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.173003 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.173505 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:36.173511 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:36.175655 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502}
W20260620 14:51:36.175978 14476 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:51:36.176162 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:36.169618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:42379 (local address 127.5.118.190:42731)
0620 14:51:36.169790 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:36.169792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:36.169991 (+ 199us) server_negotiation.cc:407] Connection header received
0620 14:51:36.170064 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:36.170065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:36.170078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:36.170158 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:36.170564 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.171299 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.172526 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.172816 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.172885 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.176032 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:42379: BlockingRecv error: recv got EOF from 127.5.118.129:42379 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
W20260620 14:51:36.238722 14480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:36.238955 14480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:36.238982 14480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:36.240201 14480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:36.240250 14480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:36.240278 14480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:36.241753 14480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.130.keytab
--principal=oryx/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:41799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=41837
--tserver_master_addrs=127.5.118.190:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:36.241964 14480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:36.242156 14480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:36.247529 14488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:36.247570 14487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:36.247534 14490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:36.248023 14480 server_base.cc:1034] running on GCE node
I20260620 14:51:36.248528 14480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:36.248837 14480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:36.250017 14480 hybrid_clock.cc:648] HybridClock initialized: now 1781967096249996 us; error 39 us; skew 500 ppm
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967096, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:36.253131 14480 init.cc:377] Logged in from keytab as oryx/127.5.118.130@KRBTEST.COM (short username oryx)
I20260620 14:51:36.253613 14480 webserver.cc:460] Webserver started at http://127.5.118.130:41837/ using document root <none> and password file <none>
I20260620 14:51:36.253839 14480 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:36.253899 14480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:36.255443 14480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:51:36.256114 14497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:36.256313 14480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:36.256431 14480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "895edd90025a403a83a58eaf5f649112"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "13a99e99ee88857afefddd7fff98c9d5"
server_key_iv: "b913a79ebe96946e367ae1552dfa411b"
server_key_version: "encryptionkey@0"
I20260620 14:51:36.256879 14480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:36.267371 14480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:36.269641 14480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:36.269829 14480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:36.270097 14480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:36.270457 14480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:36.270520 14480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:36.270596 14480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:36.270630 14480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:36.277086 14480 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:41799
I20260620 14:51:36.277172 14610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:41799 every 8 connection(s)
I20260620 14:51:36.277570 14480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:36.285701 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 14480
I20260620 14:51:36.286062 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:36931
--local_ip_for_outbound_sockets=127.5.118.131
--tserver_master_addrs=127.5.118.190:42731
--webserver_port=44879
--webserver_interface=127.5.118.131
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.118.131
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:36.286125 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:36.278473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:45167 (local address 127.5.118.190:42731)
0620 14:51:36.278546 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:36.278548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:36.278696 (+ 148us) server_negotiation.cc:407] Connection header received
0620 14:51:36.279234 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:36.279235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:36.279249 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:36.279318 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:36.280449 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.280989 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.282217 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.282463 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.282567 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.286052 (+ 3485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:45167: BlockingRecv error: recv got EOF from 127.5.118.130:45167 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:51:36.286195 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:36.278376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:45167)
0620 14:51:36.278552 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:36.278597 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:36.279108 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:36.279382 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:36.279386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:36.280009 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:36.280334 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.280341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.281096 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.281098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.282103 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.282107 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.282576 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.282578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.282725 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.283277 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:36.283286 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:36.285949 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
W20260620 14:51:36.286513 14611 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:42731 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.118.190:42731: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:36.292470 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:36.286814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:34651)
0620 14:51:36.286923 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:36.286936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:36.287005 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:36.287135 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:36.287136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:36.287314 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:36.287393 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.287396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.288147 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.288147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.289081 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.289084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.289640 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.289641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.289727 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.290205 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:36.290212 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:36.292376 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260620 14:51:36.292524 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:36.286872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:34651 (local address 127.5.118.190:42731)
0620 14:51:36.286927 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:36.286930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:36.286990 (+ 60us) server_negotiation.cc:407] Connection header received
0620 14:51:36.287070 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:36.287071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:36.287081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:36.287128 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:36.287503 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.288065 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.289211 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.289542 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.289614 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.292457 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:34651: BlockingRecv error: recv got EOF from 127.5.118.130:34651 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:36.297191 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:36.292760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:40005)
0620 14:51:36.292803 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:36.292814 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:36.292897 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:36.293058 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:36.293059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:36.293187 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:36.293218 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.293219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.293868 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.293869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.294361 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.294363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.294739 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.294740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.294800 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.295301 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:36.295305 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:36.297079 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 14:51:36.297209 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:36.292835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:40005 (local address 127.5.118.190:42731)
0620 14:51:36.292892 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:36.292893 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:36.292903 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:36.292954 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:36.292955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:36.292966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:36.293052 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:36.293303 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.293785 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.294460 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.294654 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.294738 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.297162 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:40005: BlockingRecv error: recv got EOF from 127.5.118.130:40005 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 14:51:36.297358 14611 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:51:36.376543 14614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:36.376773 14614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:36.376799 14614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:36.378021 14614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:36.378072 14614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:36.378101 14614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:36.379590 14614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/krb5kdc/oryx_127.5.118.131.keytab
--principal=oryx/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:36931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=44879
--tserver_master_addrs=127.5.118.190:42731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:36.379848 14614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:36.380098 14614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:36.385210 14619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:36.385180 14622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:36.385496 14620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:36.385476 14614 server_base.cc:1034] running on GCE node
I20260620 14:51:36.385732 14614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:36.385983 14614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:36.387146 14614 hybrid_clock.cc:648] HybridClock initialized: now 1781967096387142 us; error 37 us; skew 500 ppm
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967096, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:36.389840 14614 init.cc:377] Logged in from keytab as oryx/127.5.118.131@KRBTEST.COM (short username oryx)
I20260620 14:51:36.390306 14614 webserver.cc:460] Webserver started at http://127.5.118.131:44879/ using document root <none> and password file <none>
I20260620 14:51:36.390548 14614 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:36.390631 14614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:36.392371 14614 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:36.393080 14629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:36.393421 14614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:36.393575 14614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "2a99fccc1c6f45069f1a635aaf44eb9e"
format_stamp: "Formatted at 2026-06-20 14:51:35 on dist-test-slave-2m5f"
server_key: "11e502f11e3bea3d2f7d7e25af615713"
server_key_iv: "abfd756205667d4b5085c32541d0fbe9"
server_key_version: "encryptionkey@0"
I20260620 14:51:36.393980 14614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:36.406705 14614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:36.408977 14614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:36.409152 14614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:36.409400 14614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:36.409765 14614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:36.409830 14614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:36.409873 14614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:36.409905 14614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:36.416265 14614 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:36931
I20260620 14:51:36.416311 14742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:36931 every 8 connection(s)
I20260620 14:51:36.416685 14614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:36.422101 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 14614
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:36.424898 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:36.417459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:54273)
0620 14:51:36.417618 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:36.417659 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:36.418160 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:36.418459 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:36.418463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:36.419080 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:36.419294 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.419299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.419988 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.419989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.420949 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.420951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.421334 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.421399 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.421520 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.422123 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:36.422131 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:36.424719 (+ 2588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
W20260620 14:51:36.424896 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:36.417511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:54273 (local address 127.5.118.190:42731)
0620 14:51:36.417592 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:36.417595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:36.417751 (+ 156us) server_negotiation.cc:407] Connection header received
0620 14:51:36.418249 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:36.418250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:36.418260 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:36.418309 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:36.419390 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.419878 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.421032 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.421220 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.421280 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.424830 (+ 3550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:54273: BlockingRecv error: recv got EOF from 127.5.118.131:54273 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 14:51:36.425166 14743 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:42731 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.118.190:42731: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:36.430521 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:36.425384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:60785)
0620 14:51:36.425434 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:36.425454 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:36.425488 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:36.425616 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:36.425616 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:36.425724 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:36.425796 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.425797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.426583 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.426584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.427381 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.427384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.427927 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.427928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.428006 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.428424 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:36.428429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:36.430451 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:51:36.430596 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:36.425436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:60785 (local address 127.5.118.190:42731)
0620 14:51:36.425475 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:36.425477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:36.425483 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:51:36.425538 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:36.425538 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:36.425547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:36.425604 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:36.425872 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.426502 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.427452 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.427798 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.427925 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.430523 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:60785: BlockingRecv error: recv got EOF from 127.5.118.131:60785 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:51:36.431119 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:36.422629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.5.118.190:42731)
0620 14:51:36.422700 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:36.422702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:36.422795 (+ 93us) server_negotiation.cc:407] Connection header received
0620 14:51:36.422859 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:36.422860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:36.422870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:36.422917 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:36.423686 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.424273 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.425412 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.425704 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.425751 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.427259 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:36.427264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:36.427264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:36.427275 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:36.429629 (+ 2354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:36.429984 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:36.429988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:36.429988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:36.430110 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:36.430314 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:36.430315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:36.430315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:36.430641 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:36.430776 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:36.430910 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 14:51:36.431027 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:36 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:36.435309 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:36.430995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:60999)
0620 14:51:36.431129 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:36.431139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:36.431204 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:36.431491 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:36.431492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:36.431636 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:36.431669 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.431671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.432442 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.432447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.432919 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:36.432920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.433255 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.433256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:36.433290 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.433692 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:36.433697 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:36.435264 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:51:36.435447 14482 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:36.431034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:60999 (local address 127.5.118.190:42731)
0620 14:51:36.431071 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:36.431074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:36.431191 (+ 117us) server_negotiation.cc:407] Connection header received
0620 14:51:36.431406 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:36.431407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:36.431415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:36.431487 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:36.431769 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.432377 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.433009 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:36.433186 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:36.433249 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:36.435363 (+ 2114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:60999: BlockingRecv error: recv got EOF from 127.5.118.131:60999 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:51:36.435499 14743 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 14:51:37 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:37 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:37.181677 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:37.176613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:41073)
0620 14:51:37.176761 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:37.176774 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:37.176838 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:37.177070 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:37.177071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:37.177234 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:37.177285 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:37.177287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.178088 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.178089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:37.178615 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:37.178616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.178999 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.179001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:37.179065 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:37.179670 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:37.179676 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:37.181608 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
W20260620 14:51:37.181861 14748 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:37.176707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:41073 (local address 127.5.118.190:42731)
0620 14:51:37.176853 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:37.176854 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:37.176866 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:51:37.176948 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:37.176949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:37.176959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:37.177030 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:37.177361 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.178008 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.178689 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.178883 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.178937 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:37.181767 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:41073: BlockingRecv error: recv got EOF from 127.5.118.129:41073 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
Jun 20 14:51:37 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:37 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:37.302812 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:37.297822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:38811)
0620 14:51:37.297959 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:37.297974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:37.298033 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:37.298188 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:37.298189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:37.298372 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:37.298426 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:37.298427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.299372 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.299373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:37.300008 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:37.300010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.300421 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.300422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:37.300482 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:37.300984 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:37.300989 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:37.302743 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
W20260620 14:51:37.302909 14748 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:37.297906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:38811 (local address 127.5.118.190:42731)
0620 14:51:37.297992 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:37.297994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:37.298022 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:51:37.298091 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:37.298091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:37.298106 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:37.298167 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:37.298519 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.299296 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.300098 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.300343 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.300422 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:37.302841 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:38811: BlockingRecv error: recv got EOF from 127.5.118.130:38811 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 14:51:37 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:37 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:37.440883 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:37.435953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:55087)
0620 14:51:37.436140 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:37.436153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:37.436254 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:37.436427 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:37.436428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:37.436610 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:37.436657 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:37.436659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.437349 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.437351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:37.437891 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:37.437893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.438301 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.438302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:37.438361 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:37.438829 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:37.438834 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:37.440776 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
W20260620 14:51:37.441004 14748 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:37.436049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:55087 (local address 127.5.118.190:42731)
0620 14:51:37.436144 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:37.436147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:37.436219 (+ 72us) server_negotiation.cc:407] Connection header received
0620 14:51:37.436348 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:37.436350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:37.436364 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:37.436411 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:37.436752 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.437260 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.438003 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:37.438212 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:37.438299 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:37.440889 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:55087: BlockingRecv error: recv got EOF from 127.5.118.131:55087 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 14:51:38 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:38 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:38.186882 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:38.182159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:41753)
0620 14:51:38.182319 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:38.182329 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:38.182389 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:38.182515 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:38.182516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:38.182667 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:38.182717 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:38.182719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.183401 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.183402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:38.183981 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:38.183983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.184401 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.184402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:38.184482 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:38.184905 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:38.184909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:38.186820 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
W20260620 14:51:38.187023 14752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:38.182260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:41753 (local address 127.5.118.190:42731)
0620 14:51:38.182383 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:38.182385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:38.182424 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:51:38.182449 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:38.182450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:38.182460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:38.182509 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:38.182779 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.183315 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.184063 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.184335 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.184391 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:38.186933 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:41753: BlockingRecv error: recv got EOF from 127.5.118.129:41753 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
Jun 20 14:51:38 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:38 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:38.308007 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:38.303342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:56651)
0620 14:51:38.303501 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:38.303511 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:38.303570 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:38.303714 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:38.303715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:38.303876 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:38.303925 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:38.303927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.304632 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.304633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:38.305155 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:38.305157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.305531 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.305531 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:38.305588 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:38.306131 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:38.306136 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:38.307945 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
W20260620 14:51:38.308143 14752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:38.303423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:56651 (local address 127.5.118.190:42731)
0620 14:51:38.303502 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:38.303504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:38.303568 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:51:38.303637 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:38.303637 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:38.303649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:38.303692 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:38.303990 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.304559 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.305230 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.305419 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.305516 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:38.308060 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:56651: BlockingRecv error: recv got EOF from 127.5.118.130:56651 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 14:51:38 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:38 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:38.446250 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:38.441382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:58107)
0620 14:51:38.441634 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:38.441645 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:38.441696 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:38.441981 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:38.441981 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:38.442142 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:38.442190 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:38.442191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.442906 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.442907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:38.443439 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:38.443441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.443811 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.443812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:38.443850 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:38.444394 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:38.444399 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:38.446191 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":189,"threads_started":1}
W20260620 14:51:38.446352 14752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:38.441588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:58107 (local address 127.5.118.190:42731)
0620 14:51:38.441838 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:38.441841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:38.441851 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:38.441884 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:38.441885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:38.441904 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:38.441965 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:38.442321 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.442828 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.443539 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:38.443738 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:38.443799 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:38.446283 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:58107: BlockingRecv error: recv got EOF from 127.5.118.131:58107 (error 108)
Metrics: {"server-negotiator.queue_time_us":224}
Jun 20 14:51:39 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:39 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:39.192292 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:39.187367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:41561)
0620 14:51:39.187523 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:39.187534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:39.187585 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:39.187704 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:39.187705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:39.187861 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:39.187910 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:39.187918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.188670 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.188671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:39.189233 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:39.189235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.189616 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.189617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:39.189668 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:39.190127 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:39.190132 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:39.192151 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
W20260620 14:51:39.192344 14756 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:39.187453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:41561 (local address 127.5.118.190:42731)
0620 14:51:39.187590 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:39.187593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:39.187622 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:51:39.187643 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:39.187644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:39.187653 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:39.187702 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:39.187981 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.188583 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.189313 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.189529 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.189591 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:39.192290 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:41561: BlockingRecv error: recv got EOF from 127.5.118.129:41561 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
Jun 20 14:51:39 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:39 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:39.313190 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:39.308489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:55409)
0620 14:51:39.308626 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:39.308641 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:39.308691 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:39.308824 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:39.308825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:39.309021 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:39.309067 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:39.309068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.309736 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.309736 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:39.310277 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:39.310278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.310671 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.310671 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:39.310714 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:39.311222 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:39.311227 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:39.313129 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
W20260620 14:51:39.313304 14756 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:39.308583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:55409 (local address 127.5.118.190:42731)
0620 14:51:39.308681 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:39.308683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:39.308693 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:39.308742 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:39.308742 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:39.308752 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:39.308810 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:39.309161 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.309666 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.310355 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.310599 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.310656 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:39.313229 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:55409: BlockingRecv error: recv got EOF from 127.5.118.130:55409 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 14:51:39 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:39 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:39.452054 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:39.446975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:36327)
0620 14:51:39.447150 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:39.447163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:39.447247 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:39.447437 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:39.447438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:39.447632 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:39.447685 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:39.447687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.448502 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.448503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:39.449048 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:39.449050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.449523 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.449524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:39.449587 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:39.450078 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:39.450082 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:39.451986 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
W20260620 14:51:39.452157 14756 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:39.447043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:36327 (local address 127.5.118.190:42731)
0620 14:51:39.447137 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:39.447140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:39.447241 (+ 101us) server_negotiation.cc:407] Connection header received
0620 14:51:39.447332 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:39.447333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:39.447347 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:39.447428 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:39.447772 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.448398 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.449152 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:39.449432 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:39.449500 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:39.452087 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:36327: BlockingRecv error: recv got EOF from 127.5.118.131:36327 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 14:51:40 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:40 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:40.197939 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:40.192839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:55139)
0620 14:51:40.192977 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:40.192988 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:40.193049 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:40.193238 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:40.193240 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:40.193410 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:40.193458 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:40.193459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.194177 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.194178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:40.194764 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:40.194767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.195173 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.195174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:40.195214 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:40.195694 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:40.195699 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:40.197870 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260620 14:51:40.198086 14760 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:40.192938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:55139 (local address 127.5.118.190:42731)
0620 14:51:40.193074 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:40.193076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:40.193086 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:40.193112 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:40.193113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:40.193124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:40.193201 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:40.193559 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.194095 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.194862 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.195090 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.195164 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:40.197985 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:55139: BlockingRecv error: recv got EOF from 127.5.118.129:55139 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
Jun 20 14:51:40 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:40 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:40.321929 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:40.313772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:59005)
0620 14:51:40.314092 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:40.314104 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:40.314182 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:40.314345 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:40.314346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:40.314569 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:40.314615 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:40.314617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.315351 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.315352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:40.315948 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:40.315950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.316409 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.316410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:40.316472 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:40.319869 (+ 3397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:40.319875 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:40.321834 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":211,"threads_started":1}
W20260620 14:51:40.322005 14760 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:40.313982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:59005 (local address 127.5.118.190:42731)
0620 14:51:40.314083 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:40.314085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:40.314162 (+ 77us) server_negotiation.cc:407] Connection header received
0620 14:51:40.314245 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:40.314246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:40.314258 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:40.314339 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:40.314707 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.315247 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.316048 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.316318 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.316404 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:40.321940 (+ 5536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:59005: BlockingRecv error: recv got EOF from 127.5.118.130:59005 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 14:51:40 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:40 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:40.457383 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:40.452592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:37863)
0620 14:51:40.452746 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:40.452760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:40.452834 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:40.452993 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:40.452995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:40.453155 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:40.453205 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:40.453207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.453871 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.453872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:40.454457 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:40.454459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.454797 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.454798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:40.454846 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:40.455335 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:40.455339 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:40.457322 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
W20260620 14:51:40.457690 14760 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:40.452664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:37863 (local address 127.5.118.190:42731)
0620 14:51:40.452710 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:40.452712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:40.452811 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:51:40.452894 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:40.452894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:40.452910 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:40.452955 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:40.453278 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.453778 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.454538 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:40.454717 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:40.454769 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:40.457617 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:37863: BlockingRecv error: recv got EOF from 127.5.118.131:37863 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 14:51:41 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:41 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:41.203305 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:41.198483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:57277)
0620 14:51:41.198659 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:41.198671 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:41.198755 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:41.198945 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:41.198946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:41.199126 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:41.199204 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:41.199206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.199883 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.199884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:41.200502 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:41.200505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.200851 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.200852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:41.200907 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:41.201430 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:41.201435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:41.203247 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
W20260620 14:51:41.203477 14764 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:41.198572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:57277 (local address 127.5.118.190:42731)
0620 14:51:41.198716 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:41.198718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:41.198736 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:51:41.198832 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:41.198832 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:41.198847 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:41.198916 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:41.199285 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.199804 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.200578 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.200773 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.200830 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:41.203365 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:57277: BlockingRecv error: recv got EOF from 127.5.118.129:57277 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
Jun 20 14:51:41 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:41 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:41.328086 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:41.322617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:58273)
0620 14:51:41.322777 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:41.322792 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:41.322869 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:41.323095 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:41.323096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:41.323354 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:41.323406 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:41.323408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.324123 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.324124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:41.324764 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:41.324767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.325115 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.325116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:41.325157 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:41.325686 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:41.325691 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:41.328011 (+ 2320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
W20260620 14:51:41.328289 14764 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:41.322858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:58273 (local address 127.5.118.190:42731)
0620 14:51:41.322950 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:41.322952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:41.322962 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:41.322998 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:41.322999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:41.323010 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:41.323070 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:41.323511 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.324029 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.324848 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.325047 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.325101 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:41.328110 (+ 3009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:58273: BlockingRecv error: recv got EOF from 127.5.118.130:58273 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 14:51:41 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:41 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:41.463631 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:41.458077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:47583)
0620 14:51:41.458221 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:41.458233 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:41.458361 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:41.458574 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:41.458574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:41.458739 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:41.458785 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:41.458787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.459517 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.459518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:41.460162 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:41.460164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.460737 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.460738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:41.460804 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:41.461364 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:41.461370 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:41.463561 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260620 14:51:41.463754 14764 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:41.458153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:47583 (local address 127.5.118.190:42731)
0620 14:51:41.458237 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:41.458240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:41.458290 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:51:41.458463 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:41.458465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:41.458481 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:41.458556 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:41.458875 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.459431 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.460379 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:41.460599 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:41.460691 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:41.463654 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:47583: BlockingRecv error: recv got EOF from 127.5.118.131:47583 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 14:51:42 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:42 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:42.209405 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:42.203920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:46495)
0620 14:51:42.204079 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:42.204096 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:42.204330 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:42.204531 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:42.204532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:42.204714 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:42.204763 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:42.204764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.205437 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.205438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:42.206042 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:42.206044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.206423 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.206424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:42.206475 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:42.207059 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:42.207064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:42.209320 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
W20260620 14:51:42.209532 14768 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:42.203986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:46495 (local address 127.5.118.190:42731)
0620 14:51:42.204150 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:42.204152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:42.204162 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:42.204434 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:42.204435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:42.204445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:42.204506 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:42.204838 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.205334 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.206124 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.206315 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.206399 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:42.209450 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:46495: BlockingRecv error: recv got EOF from 127.5.118.129:46495 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
Jun 20 14:51:42 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:42 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:42.334125 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:42.328780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:38905)
0620 14:51:42.328949 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:42.328962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:42.329048 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:42.329248 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:42.329249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:42.329432 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:42.329526 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:42.329528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.330326 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.330327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:42.331137 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:42.331140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.331630 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.331631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:42.331680 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:42.332152 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:42.332157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:42.334064 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
W20260620 14:51:42.334355 14768 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:42.328859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:38905 (local address 127.5.118.190:42731)
0620 14:51:42.328918 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:42.328920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:42.329048 (+ 128us) server_negotiation.cc:407] Connection header received
0620 14:51:42.329126 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:42.329127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:42.329142 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:42.329212 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:42.329605 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.330227 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.331345 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.331547 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.331616 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:42.334288 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:38905: BlockingRecv error: recv got EOF from 127.5.118.130:38905 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Jun 20 14:51:42 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:42 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:42.468685 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:42.464097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:50027)
0620 14:51:42.464252 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:42.464265 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:42.464322 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:42.464478 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:42.464479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:42.464639 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:42.464688 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:42.464690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.465426 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.465427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:42.465959 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:42.465962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.466323 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.466324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:42.466362 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:42.466847 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:42.466852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:42.468626 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":126,"threads_started":1}
W20260620 14:51:42.468804 14768 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:42.464282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:50027 (local address 127.5.118.190:42731)
0620 14:51:42.464348 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:42.464350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:42.464369 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:51:42.464393 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:42.464394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:42.464407 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:42.464461 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:42.464793 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.465340 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.466047 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:42.466234 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:42.466310 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:42.468724 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:50027: BlockingRecv error: recv got EOF from 127.5.118.131:50027 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 14:51:43 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:43 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:43.215019 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:43.209971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:46731)
0620 14:51:43.210151 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:43.210165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:43.210243 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:43.210398 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:43.210399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:43.210561 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:43.210611 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:43.210612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.211289 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.211289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:43.211856 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:43.211858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.212254 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.212255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:43.212309 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:43.212804 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:43.212809 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:43.214955 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
W20260620 14:51:43.215090 14772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:43.210043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:46731 (local address 127.5.118.190:42731)
0620 14:51:43.210227 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:43.210229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:43.210239 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:43.210305 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:43.210306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:43.210316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:43.210368 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:43.210682 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.211210 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.211944 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.212133 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.212208 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:43.215035 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:46731: BlockingRecv error: recv got EOF from 127.5.118.129:46731 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
Jun 20 14:51:43 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:43 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:43.339600 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:43.334756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:58597)
0620 14:51:43.334963 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:43.334977 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:43.335050 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:43.335222 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:43.335223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:43.335420 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:43.335466 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:43.335468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.336148 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.336150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:43.336700 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:43.336702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.337088 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.337089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:43.337148 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:43.337653 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:43.337661 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:43.339540 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
W20260620 14:51:43.339744 14772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:43.334826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:58597 (local address 127.5.118.190:42731)
0620 14:51:43.334977 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:43.334979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:43.335030 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:51:43.335124 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:43.335125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:43.335139 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:43.335213 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:43.335553 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.336060 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.336800 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.337001 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.337087 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:43.339655 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:58597: BlockingRecv error: recv got EOF from 127.5.118.130:58597 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 14:51:43 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:43 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:43.474323 14774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:43.469226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:34447)
0620 14:51:43.469372 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:43.469385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:43.469455 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:43.469587 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:43.469588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:43.469770 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:43.469821 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:43.469823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.470512 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.470513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:43.471133 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:43.471135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.471491 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.471493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:43.471547 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:43.472160 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:43.472165 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:43.474223 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
W20260620 14:51:43.474377 14772 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:43.469287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:34447 (local address 127.5.118.190:42731)
0620 14:51:43.469360 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:43.469363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:43.469455 (+ 92us) server_negotiation.cc:407] Connection header received
0620 14:51:43.469502 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:43.469503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:43.469515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:43.469569 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:43.469901 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.470422 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.471214 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:43.471395 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:43.471464 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:43.474314 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:34447: BlockingRecv error: recv got EOF from 127.5.118.131:34447 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 14:51:44 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:44 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:44.220492 14775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:44.215625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:53643)
0620 14:51:44.215788 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:44.215801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:44.215892 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:44.216121 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:44.216122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:44.216336 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:44.216395 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:44.216397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.217048 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.217049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:44.217567 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:44.217568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.217912 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.217913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:44.217961 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:44.218464 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:44.218469 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:44.220432 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
W20260620 14:51:44.220628 14776 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:44.215739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:53643 (local address 127.5.118.190:42731)
0620 14:51:44.215857 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:44.215859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:44.215869 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:44.216010 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:44.216011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:44.216023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:44.216097 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:44.216469 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.216978 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.217641 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.217833 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.217888 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:44.220542 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:53643: BlockingRecv error: recv got EOF from 127.5.118.129:53643 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
Jun 20 14:51:44 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:44 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:44.345160 14777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:44.340123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:48431)
0620 14:51:44.340286 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:44.340300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:44.340369 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:44.340515 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:44.340516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:44.340694 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:44.340741 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:44.340742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.341637 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.341638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:44.342181 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:44.342183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.342578 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.342579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:44.342645 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:44.343116 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:44.343120 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:44.345071 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
W20260620 14:51:44.345206 14776 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:44.340241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:48431 (local address 127.5.118.190:42731)
0620 14:51:44.340309 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:44.340311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:44.340360 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:51:44.340427 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:44.340428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:44.340442 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:44.340500 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:44.340828 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.341542 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.342285 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.342490 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.342575 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:44.345158 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:48431: BlockingRecv error: recv got EOF from 127.5.118.130:48431 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 14:51:44 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:44 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:44.479494 14778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:44.474907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:39137)
0620 14:51:44.475054 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:44.475064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:44.475124 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:44.475259 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:44.475260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:44.475430 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:44.475477 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:44.475478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.476124 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.476125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:44.476758 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:44.476761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.477115 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.477115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:44.477153 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:44.477665 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:44.477670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:44.479383 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
W20260620 14:51:44.479573 14776 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:44.474979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:39137 (local address 127.5.118.190:42731)
0620 14:51:44.475055 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:44.475057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:44.475127 (+ 70us) server_negotiation.cc:407] Connection header received
0620 14:51:44.475168 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:44.475169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:44.475181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:44.475239 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:44.475551 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.476050 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.476866 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:44.477048 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:44.477104 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:44.479484 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:39137: BlockingRecv error: recv got EOF from 127.5.118.131:39137 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 14:51:45 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:45 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:45.226274 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:45.221082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:41537)
0620 14:51:45.221249 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:45.221260 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:45.221314 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:45.221511 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:45.221512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:45.221686 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:45.221736 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:45.221738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.222456 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.222457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:45.223025 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:45.223028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.223509 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.223511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:45.223587 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:45.224148 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:45.224153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:45.226209 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
W20260620 14:51:45.226444 14780 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:45.221227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:41537 (local address 127.5.118.190:42731)
0620 14:51:45.221365 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:45.221366 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:45.221375 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:51:45.221402 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:45.221402 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:45.221412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:45.221489 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:45.221807 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.222344 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.223119 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.223337 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.223395 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:45.226351 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:41537: BlockingRecv error: recv got EOF from 127.5.118.129:41537 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
Jun 20 14:51:45 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:45 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:45.355556 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:45.345876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:45439)
0620 14:51:45.346120 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:45.346135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:45.346318 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:45.346515 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:45.346516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:45.346714 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:45.346772 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:45.346774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.347464 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.347465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:45.347992 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:45.347994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.348462 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.348463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:45.348528 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:45.352594 (+ 4066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:45.352607 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:45.355443 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":131,"threads_started":1}
W20260620 14:51:45.355657 14780 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:45.346030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:45439 (local address 127.5.118.190:42731)
0620 14:51:45.346135 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:45.346138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:45.346293 (+ 155us) server_negotiation.cc:407] Connection header received
0620 14:51:45.346418 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:45.346419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:45.346440 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:45.346506 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:45.346866 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.347376 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.348095 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.348369 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.348461 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:45.355584 (+ 7123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:45439: BlockingRecv error: recv got EOF from 127.5.118.130:45439 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 14:51:45 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:45 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:45.485791 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:45.480163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:56369)
0620 14:51:45.480378 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:45.480391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:45.480465 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:45.480634 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:45.480635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:45.480805 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:45.480860 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:45.480862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.482024 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.482025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:45.482608 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:45.482610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.482990 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.482991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:45.483033 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:45.483540 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:45.483545 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:45.485729 (+ 2184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
W20260620 14:51:45.485977 14780 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:45.480348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:56369 (local address 127.5.118.190:42731)
0620 14:51:45.480414 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:45.480419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:45.480448 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:51:45.480536 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:45.480537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:45.480548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:45.480602 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:45.481007 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.481933 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.482702 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:45.482916 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:45.482981 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:45.485879 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:56369: BlockingRecv error: recv got EOF from 127.5.118.131:56369 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 14:51:46 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:46 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:46.232791 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:46.226880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:42935)
0620 14:51:46.227049 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:46.227093 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:46.227185 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:46.227514 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:46.227515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:46.227857 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:46.227977 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:46.227979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.228948 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.228949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:46.229620 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:46.229623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.230092 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.230093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:46.230146 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:46.230643 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:46.230648 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:46.232727 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
W20260620 14:51:46.233084 14784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:46.226976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:42935 (local address 127.5.118.190:42731)
0620 14:51:46.227206 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:46.227209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:46.227222 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:51:46.227361 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:46.227363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:46.227382 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:46.227487 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:46.228090 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.228854 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.229755 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.229995 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.230063 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:46.232884 (+ 2821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:42935: BlockingRecv error: recv got EOF from 127.5.118.129:42935 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":62,"threads_started":1}
Jun 20 14:51:46 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:46 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:46.362200 14785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:46.356376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:48571)
0620 14:51:46.356525 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:46.356543 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:46.356581 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:46.356808 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:46.356810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:46.357101 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:46.357172 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:46.357175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.358028 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.358029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:46.358672 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:46.358674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.359097 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.359099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:46.359148 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:46.359735 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:46.359741 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:46.362120 (+ 2379us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
W20260620 14:51:46.362351 14784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:46.356475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:48571 (local address 127.5.118.190:42731)
0620 14:51:46.356540 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:46.356543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:46.356675 (+ 132us) server_negotiation.cc:407] Connection header received
0620 14:51:46.356706 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:46.356707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:46.356719 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:46.356782 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:46.357275 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.357938 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.358759 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.359012 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.359081 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:46.362249 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:48571: BlockingRecv error: recv got EOF from 127.5.118.130:48571 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 14:51:46 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:46 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:46.491647 14786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:46.486417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:59893)
0620 14:51:46.486587 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:46.486606 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:46.486689 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:46.486931 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:46.486933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:46.487120 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:46.487167 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:46.487168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.487894 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.487895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:46.488466 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:46.488468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.488885 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.488886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:46.488944 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:46.489468 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:46.489481 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:46.491547 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
W20260620 14:51:46.491703 14784 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:46.486611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:59893 (local address 127.5.118.190:42731)
0620 14:51:46.486691 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:46.486693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:46.486703 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:46.486823 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:46.486824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:46.486840 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:46.486910 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:46.487269 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.487777 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.488563 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:46.488773 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:46.488886 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:46.491628 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:59893: BlockingRecv error: recv got EOF from 127.5.118.131:59893 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 14:51:47 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:47 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:47.238816 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:47.233491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:43695)
0620 14:51:47.233696 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:47.233711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:47.233794 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:47.233953 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:47.233954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:47.234129 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:47.234180 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:47.234182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.234975 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.234976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:47.235558 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:47.235560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.235955 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.235956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:47.235996 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:47.236589 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:47.236594 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:47.238650 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":107,"threads_started":1}
W20260620 14:51:47.238829 14788 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:47.233598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:43695 (local address 127.5.118.190:42731)
0620 14:51:47.233707 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:47.233712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:47.233775 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:51:47.233856 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:47.233857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:47.233868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:47.233922 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:47.234356 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.234880 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.235683 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.235881 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.235942 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:47.238755 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:43695: BlockingRecv error: recv got EOF from 127.5.118.129:43695 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
Jun 20 14:51:47 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:47 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:47.368844 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:47.362918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:56623)
0620 14:51:47.363197 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:47.363221 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:47.363310 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:47.363506 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:47.363507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:47.363760 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:47.363824 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:47.363827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.364799 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.364801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:47.365461 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:47.365464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.365929 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.365930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:47.366001 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:47.366535 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:47.366541 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:47.368740 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":136,"threads_started":1}
W20260620 14:51:47.368934 14788 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:47.363059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:56623 (local address 127.5.118.190:42731)
0620 14:51:47.363191 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:47.363291 (+ 100us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:47.363312 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:51:47.363384 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:47.363385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:47.363407 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:47.363497 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:47.363933 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.364673 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.365573 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.365808 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.365916 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:47.368870 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:56623: BlockingRecv error: recv got EOF from 127.5.118.130:56623 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 14:51:47 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:47 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:51:47.497619 14788 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:47.492296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:51305 (local address 127.5.118.190:42731)
0620 14:51:47.492397 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:47.492399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:47.492534 (+ 135us) server_negotiation.cc:407] Connection header received
0620 14:51:47.492650 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:47.492651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:47.492667 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:47.492758 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:47.493086 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.493590 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.494328 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.494524 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.494615 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:47.497564 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:51305: BlockingRecv error: recv got EOF from 127.5.118.131:51305 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:51:47.497618 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:47.492266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:51305)
0620 14:51:47.492451 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:47.492462 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:47.492556 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:47.492765 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:47.492766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:47.492956 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:47.493000 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:47.493002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.493680 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.493681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:47.494230 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:47.494232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:47.494615 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:47.494616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:47.494679 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:47.495201 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:47.495206 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:47.497491 (+ 2285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":101,"threads_started":1}
Jun 20 14:51:48 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:48 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:48.245296 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:48.239540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:58919)
0620 14:51:48.239692 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:48.239706 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:48.239776 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:48.240109 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:48.240110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:48.240347 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:48.240406 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:48.240408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.241123 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.241124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:48.241746 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:48.241749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.242320 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.242321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:48.242401 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:48.243067 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:48.243073 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:48.245150 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
W20260620 14:51:48.245369 14792 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:48.239672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:58919 (local address 127.5.118.190:42731)
0620 14:51:48.239852 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:48.239855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:48.239867 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:51:48.239961 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:48.239962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:48.239976 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:48.240055 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:48.240504 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.241023 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.241855 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.242182 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.242265 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:48.245316 (+ 3051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:58919: BlockingRecv error: recv got EOF from 127.5.118.129:58919 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":109,"threads_started":1}
Jun 20 14:51:48 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:48 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:48.374862 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:48.369507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:54317)
0620 14:51:48.369681 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:48.369696 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:48.369862 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:48.370022 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:48.370023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:48.370435 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:48.370485 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:48.370488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.371238 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.371239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:48.371785 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:48.371787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.372165 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.372166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:48.372268 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:48.372808 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:48.372813 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:48.374768 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":112,"threads_started":1}
W20260620 14:51:48.374995 14792 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:48.369649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:54317 (local address 127.5.118.190:42731)
0620 14:51:48.369713 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:48.369715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:48.369844 (+ 129us) server_negotiation.cc:407] Connection header received
0620 14:51:48.369927 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:48.369928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:48.369942 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:48.370017 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:48.370611 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.371124 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.371886 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.372084 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.372166 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:48.374927 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:54317: BlockingRecv error: recv got EOF from 127.5.118.130:54317 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 14:51:48 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:48 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:48.502686 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:48.498108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:54429)
0620 14:51:48.498231 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:48.498241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:48.498299 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:48.498426 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:48.498427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:48.498585 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:48.498630 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:48.498631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.499332 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.499333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:48.499848 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:48.499849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.500247 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.500247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:48.500296 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:48.500761 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:48.500765 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:48.502627 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
W20260620 14:51:48.502851 14792 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:48.498155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:54429 (local address 127.5.118.190:42731)
0620 14:51:48.498202 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:48.498204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:48.498341 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:51:48.498364 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:48.498364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:48.498378 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:48.498420 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:48.498718 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.499256 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.499945 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:48.500134 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:48.500264 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:48.502764 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:54429: BlockingRecv error: recv got EOF from 127.5.118.131:54429 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 14:51:49 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:49 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:49.251817 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:49.245999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:37469)
0620 14:51:49.246169 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:49.246184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:49.246370 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:49.246521 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:49.246522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:49.246698 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:49.246758 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:49.246760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.247546 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.247548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:49.248135 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:49.248138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.248679 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.248680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:49.248736 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:49.249322 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:49.249327 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:49.251738 (+ 2411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":55,"threads_started":1}
W20260620 14:51:49.251982 14796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:49.246135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:37469 (local address 127.5.118.190:42731)
0620 14:51:49.246304 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:49.246306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:49.246368 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:51:49.246422 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:49.246423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:49.246433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:49.246493 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:49.246883 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.247437 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.248321 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.248561 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.248639 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:49.251870 (+ 3231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:37469: BlockingRecv error: recv got EOF from 127.5.118.129:37469 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
Jun 20 14:51:49 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:49 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:49.381078 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:49.375484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:48277)
0620 14:51:49.375629 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:49.375642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:49.375704 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:49.375941 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:49.375942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:49.376113 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:49.376161 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:49.376163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.377097 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.377098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:49.377672 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:49.377675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.378066 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.378067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:49.378116 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:49.378688 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:49.378694 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:49.380962 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
W20260620 14:51:49.381122 14796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:49.375679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:48277 (local address 127.5.118.190:42731)
0620 14:51:49.375748 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:49.375750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:49.375760 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:49.375800 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:49.375801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:49.375811 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:49.375911 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:49.376413 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.376972 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.377762 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.377971 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.378030 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:49.381059 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:48277: BlockingRecv error: recv got EOF from 127.5.118.130:48277 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:51:49 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:49 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:49.512441 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:49.503237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:48971)
0620 14:51:49.503539 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:49.503552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:49.503661 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:49.503824 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:49.503825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:49.504030 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:49.504081 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:49.504083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.504865 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.504866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:49.505504 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:49.505507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.505957 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.505958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:49.506015 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:49.510047 (+ 4032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:49.510053 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:49.512319 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":110,"threads_started":1}
W20260620 14:51:49.512488 14796 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:49.503361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:48971 (local address 127.5.118.190:42731)
0620 14:51:49.503539 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:49.503542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:49.503641 (+ 99us) server_negotiation.cc:407] Connection header received
0620 14:51:49.503723 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:49.503725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:49.503743 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:49.503817 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:49.504201 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.504722 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.505598 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:49.505795 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:49.505885 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:49.512417 (+ 6532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:48971: BlockingRecv error: recv got EOF from 127.5.118.131:48971 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Jun 20 14:51:50 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:50 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:50.257356 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:50.252516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:54627)
0620 14:51:50.252675 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:50.252691 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:50.252769 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:50.252933 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:50.252934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:50.253116 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:50.253164 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:50.253166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.253871 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.253872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:50.254404 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:50.254406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.254804 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.254805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:50.254873 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:50.255377 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:50.255382 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:50.257238 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
W20260620 14:51:50.257515 14800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:50.252564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:54627 (local address 127.5.118.190:42731)
0620 14:51:50.252709 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:50.252711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:50.252777 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:51:50.252841 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:50.252842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:50.252856 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:50.252921 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:50.253258 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.253779 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.254512 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.254716 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.254801 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:50.257422 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:54627: BlockingRecv error: recv got EOF from 127.5.118.129:54627 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
Jun 20 14:51:50 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:50 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:50.390938 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:50.381946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:46059)
0620 14:51:50.382208 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:50.382220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:50.382302 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:50.382476 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:50.382477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:50.382661 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:50.382708 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:50.382710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.383434 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.383435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:50.384025 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:50.384028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.384478 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.384479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:50.384539 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:50.388552 (+ 4013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:50.388558 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:50.390851 (+ 2293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
W20260620 14:51:50.391104 14800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:50.382081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:46059 (local address 127.5.118.190:42731)
0620 14:51:50.382162 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:50.382164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:50.382344 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:51:50.382382 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:50.382383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:50.382396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:50.382441 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:50.382803 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.383332 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.384126 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.384363 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.384424 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:50.391022 (+ 6598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:46059: BlockingRecv error: recv got EOF from 127.5.118.130:46059 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 14:51:50 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:50 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.131@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:50.518657 14802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:50.513111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.131:33789)
0620 14:51:50.513248 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:50.513268 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:50.513343 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:50.513507 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:50.513508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:50.513736 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:50.513786 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:50.513788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.514572 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.514573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:50.515206 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:50.515209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.515647 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.515648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:50.515733 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:50.516383 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:50.516388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:50.518554 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
W20260620 14:51:50.518709 14800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:50.513189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:33789 (local address 127.5.118.190:42731)
0620 14:51:50.513261 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:50.513263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:50.513341 (+ 78us) server_negotiation.cc:407] Connection header received
0620 14:51:50.513402 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:50.513403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:50.513417 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:50.513475 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:50.513894 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.514458 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.515359 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:50.515554 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:50.515622 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:50.518640 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.131:33789: BlockingRecv error: recv got EOF from 127.5.118.131:33789 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 14:51:51 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:51 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.129@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:51.263188 14803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.258009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.129:42871)
0620 14:51:51.258173 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:51.258184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:51.258246 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:51.258395 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:51.258395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:51.258555 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:51.258602 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:51.258603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.259302 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.259304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:51.260018 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:51.260021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.260613 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.260614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:51.260684 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:51.261232 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:51.261238 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:51.263125 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
W20260620 14:51:51.263329 14804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:51.258142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:42871 (local address 127.5.118.190:42731)
0620 14:51:51.258249 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:51.258251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:51.258269 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:51:51.258304 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:51.258305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:51.258315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:51.258376 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:51.258676 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.259178 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.260192 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.260512 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.260590 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:51.263253 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.129:42871: BlockingRecv error: recv got EOF from 127.5.118.129:42871 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":56,"threads_started":1}
Jun 20 14:51:51 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:51:51 dist-test-slave-2m5f krb5kdc[13745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.118.130@KRBTEST.COM for oryx/127.5.118.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:51:51.396504 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.391554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42731 (local address 127.5.118.130:38331)
0620 14:51:51.391697 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:51.391708 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:51.391810 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:51.391967 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:51.391967 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:51.392149 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:51.392221 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:51.392223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.393019 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.393020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:51.393558 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:51.393560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.393946 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.393948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:51.394007 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:51.394459 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:51.394463 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:51.396443 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.118.190:42731: Server oryx/127.5.118.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
W20260620 14:51:51.396636 14804 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:51.391599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:38331 (local address 127.5.118.190:42731)
0620 14:51:51.391687 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:51.391690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:51.391787 (+ 97us) server_negotiation.cc:407] Connection header received
0620 14:51:51.391884 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:51.391885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:51.391899 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:51.391950 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:51.392342 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.392930 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.393653 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.393855 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.393945 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:51.396554 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.118.130:38331: BlockingRecv error: recv got EOF from 127.5.118.130:38331 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:51:51.422748 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 14328
I20260620 14:51:51.427403 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 14480
I20260620 14:51:51.431881 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 14614
I20260620 14:51:51.436400 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 14258
2026-06-20T14:51:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17511 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:51:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:51Z Disabled control of system clock
I20260620 14:51:51.464068 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:45795
--webserver_interface=127.5.118.190
--webserver_port=0
--builtin_ntp_servers=127.5.118.148:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:45795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:51:51.546348 14809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:51.546577 14809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:51.546597 14809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:51.547878 14809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:51.547931 14809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:51.547945 14809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:51.547957 14809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:51.547971 14809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:51.549464 14809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:45795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:45795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:51.549744 14809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:51.550071 14809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:51:51.555507 14809 server_base.cc:1034] running on GCE node
W20260620 14:51:51.555531 14815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.555474 14814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.555474 14817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:51.556053 14809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:51.556391 14809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:51.557567 14809 hybrid_clock.cc:648] HybridClock initialized: now 1781967111557552 us; error 29 us; skew 500 ppm
I20260620 14:51:51.558786 14809 webserver.cc:460] Webserver started at http://127.5.118.190:38305/ using document root <none> and password file <none>
I20260620 14:51:51.559036 14809 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:51.559103 14809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:51.559257 14809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:51.560483 14809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "56d678f4da5f425385a7557d555dac45"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "a94b2aebdcdf3a610a13929d4f349e8e"
server_key_iv: "3a243d02dab3f40c76a0a4633856c5ab"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.560847 14809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "56d678f4da5f425385a7557d555dac45"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "a94b2aebdcdf3a610a13929d4f349e8e"
server_key_iv: "3a243d02dab3f40c76a0a4633856c5ab"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.562191 14809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:51.562966 14823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.563141 14809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:51.563251 14809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "56d678f4da5f425385a7557d555dac45"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "a94b2aebdcdf3a610a13929d4f349e8e"
server_key_iv: "3a243d02dab3f40c76a0a4633856c5ab"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.563364 14809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:51.576455 14809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:51.576800 14809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:51.576982 14809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:51.581131 14809 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:45795
I20260620 14:51:51.581164 14875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:45795 every 8 connection(s)
I20260620 14:51:51.581553 14809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:51.582257 14876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:51.584960 14876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45: Bootstrap starting.
I20260620 14:51:51.585575 14876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:51.585894 14876 log.cc:824] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:51.586608 14876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45: No bootstrap required, opened a new log
I20260620 14:51:51.588397 14876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d678f4da5f425385a7557d555dac45" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45795 } }
I20260620 14:51:51.588543 14876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:51.588582 14876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d678f4da5f425385a7557d555dac45, State: Initialized, Role: FOLLOWER
I20260620 14:51:51.588752 14876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d678f4da5f425385a7557d555dac45" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45795 } }
I20260620 14:51:51.588774 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 14809
I20260620 14:51:51.588858 14876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:51.588883 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:51.588897 14876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:51.588944 14876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:51.589097 5594 external_mini_cluster.cc:1428] Setting key 836100c1f6f5104b2039b8b7651eb4a4
I20260620 14:51:51.589819 14876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d678f4da5f425385a7557d555dac45" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45795 } }
I20260620 14:51:51.589927 14876 leader_election.cc:302] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d678f4da5f425385a7557d555dac45; no voters:
I20260620 14:51:51.590111 14876 leader_election.cc:288] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:51.590138 14881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:51.590307 14881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [term 1 LEADER]: Becoming Leader. State: Replica: 56d678f4da5f425385a7557d555dac45, State: Running, Role: LEADER
I20260620 14:51:51.590372 14876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:51.590464 14881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d678f4da5f425385a7557d555dac45" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45795 } }
I20260620 14:51:51.590930 14881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56d678f4da5f425385a7557d555dac45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d678f4da5f425385a7557d555dac45" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45795 } } }
I20260620 14:51:51.591076 14881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:51.591248 14881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d678f4da5f425385a7557d555dac45. Latest consensus state: current_term: 1 leader_uuid: "56d678f4da5f425385a7557d555dac45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d678f4da5f425385a7557d555dac45" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45795 } } }
I20260620 14:51:51.591352 14881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:51.592242 14890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:51.593009 14890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:51.594707 14890 catalog_manager.cc:1347] Generated new cluster ID: f9b53694fb57415685139d5cb1e1295c
I20260620 14:51:51.594760 14890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:51.596071 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.589665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.5.118.190:45795)
0620 14:51:51.589915 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:51.589925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:51.589961 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:51:51.590088 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:51.590112 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:51.590194 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:51.590355 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:51.590595 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.592657 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.593710 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.594259 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.594429 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:51.595234 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:51.595238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:51.595238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:51.595448 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:51.595489 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:51.595504 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:51.595564 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:51.595773 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 14:51:51.595864 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260620 14:51:51.603317 14890 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:51.604051 14890 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:51.619107 14890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d678f4da5f425385a7557d555dac45: Generated new TSK 0
I20260620 14:51:51.619372 14890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:51:51.626500 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:45795
--builtin_ntp_servers=127.5.118.148:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:51:51.713850 14900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:51.714085 14900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:51.714111 14900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:51.715481 14900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:51.715533 14900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:51.715564 14900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:51.717249 14900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:45795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:51.717549 14900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:51.717897 14900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:51.724200 14906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.724134 14908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.724335 14905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:51.724524 14900 server_base.cc:1034] running on GCE node
I20260620 14:51:51.724808 14900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:51.725082 14900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:51.726269 14900 hybrid_clock.cc:648] HybridClock initialized: now 1781967111726240 us; error 39 us; skew 500 ppm
I20260620 14:51:51.727526 14900 webserver.cc:460] Webserver started at http://127.5.118.129:32945/ using document root <none> and password file <none>
I20260620 14:51:51.727815 14900 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:51.727900 14900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:51.728066 14900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:51.729359 14900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "d93da6e3cf334dfc992363f67d2062f3"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "90af3b7727069005f970ee908ad1796a"
server_key_iv: "b690dba15dad513e55f3b3216c6b73f1"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.729753 14900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "d93da6e3cf334dfc992363f67d2062f3"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "90af3b7727069005f970ee908ad1796a"
server_key_iv: "b690dba15dad513e55f3b3216c6b73f1"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.731207 14900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:51.732046 14914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.732265 14900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:51.732369 14900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "d93da6e3cf334dfc992363f67d2062f3"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "90af3b7727069005f970ee908ad1796a"
server_key_iv: "b690dba15dad513e55f3b3216c6b73f1"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.732457 14900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:51.744606 14900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:51.744967 14900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:51.745128 14900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:51.745379 14900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:51.745748 14900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:51.745808 14900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.745853 14900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:51.745886 14900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.751957 14900 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:44293
I20260620 14:51:51.752000 15027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:44293 every 8 connection(s)
I20260620 14:51:51.752446 14900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:51.758121 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.753232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:43537 (local address 127.5.118.190:45795)
0620 14:51:51.753304 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:51.753306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:51.753587 (+ 281us) server_negotiation.cc:407] Connection header received
0620 14:51:51.754065 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:51.754066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:51.754078 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:51.754130 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:51.754873 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.755409 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.756709 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.756914 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.756973 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:51.757868 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:51.757869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:51.757869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:51.757876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:51.757889 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:51.757893 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:51.757938 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:51.758028 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 14:51:51.758056 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:51:51.758383 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.753242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:45795 (local address 127.5.118.129:43537)
0620 14:51:51.753442 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:51.753479 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:51.753975 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:51.754274 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:51.754278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:51.754287 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:51.754788 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:51.754794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.755523 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.755524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:51.756617 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:51.756620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.757020 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.757022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:51.757155 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:51.757702 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:51.757702 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:51.757733 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:51.757758 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:51.757952 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:51.757957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:51.757960 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:51.758015 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:51:51.758101 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260620 14:51:51.759025 15028 heartbeater.cc:347] Connected to a master server at 127.5.118.190:45795
I20260620 14:51:51.759137 15028 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:51.759351 15028 heartbeater.cc:511] Master 127.5.118.190:45795 requested a full tablet report, sending...
I20260620 14:51:51.759881 14840 ts_manager.cc:194] Registered new tserver with Master: d93da6e3cf334dfc992363f67d2062f3 (127.5.118.129:44293)
I20260620 14:51:51.760599 14840 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.129:43537
I20260620 14:51:51.761683 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 14900
I20260620 14:51:51.761785 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:51.762001 5594 external_mini_cluster.cc:1428] Setting key ba85115d0d2cba2fd35ac4baa0fb5340
I20260620 14:51:51.763005 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:45795
--builtin_ntp_servers=127.5.118.148:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:51:51.844552 15031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:51.844753 15031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:51.844774 15031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:51.845947 15031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:51.845986 15031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:51.846011 15031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:51.847349 15031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:45795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:51.847613 15031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:51.847929 15031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:51.853029 15036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.853029 15039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.853020 15037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:51.853485 15031 server_base.cc:1034] running on GCE node
I20260620 14:51:51.853744 15031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:51.854017 15031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:51.855185 15031 hybrid_clock.cc:648] HybridClock initialized: now 1781967111855178 us; error 43 us; skew 500 ppm
I20260620 14:51:51.856570 15031 webserver.cc:460] Webserver started at http://127.5.118.130:35713/ using document root <none> and password file <none>
I20260620 14:51:51.856830 15031 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:51.856905 15031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:51.857057 15031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:51.858139 15031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "ddd3be1d2d834688a82e53968a6b9c54"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "8cd5b59d05f2d2cac3f2347044a215ce"
server_key_iv: "43749fea62b056a9914c40f3a07c27c2"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.858516 15031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "ddd3be1d2d834688a82e53968a6b9c54"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "8cd5b59d05f2d2cac3f2347044a215ce"
server_key_iv: "43749fea62b056a9914c40f3a07c27c2"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.859819 15031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:51.860632 15045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.860822 15031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.860914 15031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "ddd3be1d2d834688a82e53968a6b9c54"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "8cd5b59d05f2d2cac3f2347044a215ce"
server_key_iv: "43749fea62b056a9914c40f3a07c27c2"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.860999 15031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:51.874578 15031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:51.874929 15031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:51.875092 15031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:51.875350 15031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:51.875700 15031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:51.875758 15031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.875799 15031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:51.875833 15031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.881918 15031 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:35593
I20260620 14:51:51.882118 15158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:35593 every 8 connection(s)
I20260620 14:51:51.882390 15031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:51.888042 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.883332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:56715 (local address 127.5.118.190:45795)
0620 14:51:51.883406 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:51.883408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:51.883633 (+ 225us) server_negotiation.cc:407] Connection header received
0620 14:51:51.884213 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:51.884214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:51.884226 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:51.884305 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:51.885038 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.885526 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.886626 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.886821 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.886877 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:51.887746 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:51.887747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:51.887747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:51.887753 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:51.887766 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:51.887770 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:51.887827 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:51.887935 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:51:51.887961 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:51:51.888279 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 15031
I20260620 14:51:51.888383 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:51.888463 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.883251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:45795 (local address 127.5.118.130:56715)
0620 14:51:51.883486 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:51.883533 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:51.884076 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:51.884390 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:51.884394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:51.884403 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:51.884954 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:51.884959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.885617 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.885618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:51.886537 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:51.886539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:51.886928 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:51.886929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:51.887060 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:51.887600 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:51.887600 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:51.887630 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:51.887654 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:51.887852 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:51.887857 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:51.887859 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:51:51.887920 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 14:51:51.888008 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":121,"threads_started":1}
I20260620 14:51:51.888618 5594 external_mini_cluster.cc:1428] Setting key a6ff9fb72fd8f8e0e9d81e5a6e883fe4
I20260620 14:51:51.888974 15159 heartbeater.cc:347] Connected to a master server at 127.5.118.190:45795
I20260620 14:51:51.889070 15159 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:51.889304 15159 heartbeater.cc:511] Master 127.5.118.190:45795 requested a full tablet report, sending...
I20260620 14:51:51.889700 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:45795
--builtin_ntp_servers=127.5.118.148:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:51:51.889717 14840 ts_manager.cc:194] Registered new tserver with Master: ddd3be1d2d834688a82e53968a6b9c54 (127.5.118.130:35593)
I20260620 14:51:51.890235 14840 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.130:56715
W20260620 14:51:51.974952 15162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:51.975171 15162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:51.975193 15162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:51.976397 15162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:51.976437 15162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:51.976462 15162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:51.977778 15162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:45795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:51.977973 15162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:51.978235 15162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:51.983511 15168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.983520 15170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.983520 15167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:51.983551 15162 server_base.cc:1034] running on GCE node
I20260620 14:51:51.984046 15162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:51.984356 15162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:51.985517 15162 hybrid_clock.cc:648] HybridClock initialized: now 1781967111985510 us; error 30 us; skew 500 ppm
I20260620 14:51:51.986792 15162 webserver.cc:460] Webserver started at http://127.5.118.131:35893/ using document root <none> and password file <none>
I20260620 14:51:51.987044 15162 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:51.987116 15162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:51.987319 15162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:51.988548 15162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "437aec8efdd0432c8d7ae9fb9cee3d0b"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "85bc37c16998c51910fbc417a88315ca"
server_key_iv: "b061368ad9157b3265d2ee096cf9657c"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.988935 15162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "437aec8efdd0432c8d7ae9fb9cee3d0b"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "85bc37c16998c51910fbc417a88315ca"
server_key_iv: "b061368ad9157b3265d2ee096cf9657c"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.990324 15162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:51.991060 15176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.991227 15162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:51.991329 15162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "437aec8efdd0432c8d7ae9fb9cee3d0b"
format_stamp: "Formatted at 2026-06-20 14:51:51 on dist-test-slave-2m5f"
server_key: "85bc37c16998c51910fbc417a88315ca"
server_key_iv: "b061368ad9157b3265d2ee096cf9657c"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.991390 15162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:52.006026 15162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:52.006317 15162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:52.006421 15162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:52.006601 15162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:52.006882 15162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:52.006912 15162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.006934 15162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:52.006946 15162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.012549 15162 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:42181
I20260620 14:51:52.012631 15289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:42181 every 8 connection(s)
I20260620 14:51:52.012998 15162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:52.014839 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 15162
I20260620 14:51:52.014919 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:52.015121 5594 external_mini_cluster.cc:1428] Setting key af961deb43b2ef333ad1ee3d82a93fe0
I20260620 14:51:52.018966 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.013811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:43317 (local address 127.5.118.190:45795)
0620 14:51:52.013884 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:52.013886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:52.014146 (+ 260us) server_negotiation.cc:407] Connection header received
0620 14:51:52.014753 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:52.014755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:52.014769 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:52.014830 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:52.015731 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:52.016448 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:52.017512 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:52.017733 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:52.017828 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:52.018672 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:52.018674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:52.018674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:52.018679 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:52.018695 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:52.018699 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:52.018743 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:52.018853 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 14:51:52.018884 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:51:52.019207 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.013793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:45795 (local address 127.5.118.131:43317)
0620 14:51:52.014020 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.014054 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:52.014610 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:52.014909 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:52.014913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:52.014924 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:52.015652 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:52.015660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:52.016557 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:52.016558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:52.017408 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:52.017410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:52.017840 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:52.017841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:52.017991 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:52.018518 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:52.018519 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:52.018550 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:52.018573 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:52.018753 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:52.018758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:52.018761 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:52.018856 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:51:52.018959 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":99,"threads_started":1}
I20260620 14:51:52.019706 15290 heartbeater.cc:347] Connected to a master server at 127.5.118.190:45795
I20260620 14:51:52.019794 15290 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:52.019992 15290 heartbeater.cc:511] Master 127.5.118.190:45795 requested a full tablet report, sending...
I20260620 14:51:52.020416 14840 ts_manager.cc:194] Registered new tserver with Master: 437aec8efdd0432c8d7ae9fb9cee3d0b (127.5.118.131:42181)
I20260620 14:51:52.020787 14840 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.131:43317
I20260620 14:51:52.027148 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:51:52.029147 14880 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:52.028605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.5.118.190:45795)
0620 14:51:52.028699 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:52.028701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:52.028793 (+ 92us) server_negotiation.cc:407] Connection header received
0620 14:51:52.028871 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:52.028872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:52.028885 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:52.028930 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:52.029076 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42812: BlockingRecv error: recv got EOF from 127.0.0.1:42812 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:51:52.029215 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 14900
I20260620 14:51:52.033567 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 15031
I20260620 14:51:52.037900 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 15162
I20260620 14:51:52.042171 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 14809
2026-06-20T14:51:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (605 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:52Z Disabled control of system clock
I20260620 14:51:52.069660 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:34511
--webserver_interface=127.5.118.190
--webserver_port=0
--builtin_ntp_servers=127.5.118.148:38307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:34511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:51:52.161429 15301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:52.161666 15301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:52.161691 15301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:52.163101 15301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:52.163156 15301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:52.163172 15301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:52.163185 15301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:52.163199 15301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:52.164790 15301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:38307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:34511
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:52.165025 15301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:52.165307 15301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:52.170910 15306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.170939 15309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.170938 15307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:52.171115 15301 server_base.cc:1034] running on GCE node
I20260620 14:51:52.171478 15301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:52.171823 15301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:52.173039 15301 hybrid_clock.cc:648] HybridClock initialized: now 1781967112173015 us; error 41 us; skew 500 ppm
I20260620 14:51:52.174436 15301 webserver.cc:460] Webserver started at http://127.5.118.190:37617/ using document root <none> and password file <none>
I20260620 14:51:52.174751 15301 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:52.174805 15301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:52.174953 15301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:52.176484 15301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "bd508df4fdd3436dbab409f6d46f7a99"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "be475f41140da699fb707750419e1c90"
server_key_iv: "6e08611bc5b851b8ebc3bb385bc1014e"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.176892 15301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "bd508df4fdd3436dbab409f6d46f7a99"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "be475f41140da699fb707750419e1c90"
server_key_iv: "6e08611bc5b851b8ebc3bb385bc1014e"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.178403 15301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:52.179271 15315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.179585 15301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:52.179674 15301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "bd508df4fdd3436dbab409f6d46f7a99"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "be475f41140da699fb707750419e1c90"
server_key_iv: "6e08611bc5b851b8ebc3bb385bc1014e"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.179780 15301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:52.183488 15301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:52.183780 15301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:52.183938 15301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:52.187916 15301 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:34511
I20260620 14:51:52.187947 15367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:34511 every 8 connection(s)
I20260620 14:51:52.188400 15301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:52.189086 15368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:52.192060 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99: Bootstrap starting.
I20260620 14:51:52.192710 15368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:52.193040 15368 log.cc:824] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:52.193745 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99: No bootstrap required, opened a new log
I20260620 14:51:52.194108 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 15301
I20260620 14:51:52.194268 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:52.194470 5594 external_mini_cluster.cc:1428] Setting key 946d756b3e278cb3d15a5d7a6bb436ba
I20260620 14:51:52.196089 15368 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd508df4fdd3436dbab409f6d46f7a99" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34511 } }
I20260620 14:51:52.196295 15368 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:52.196337 15368 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd508df4fdd3436dbab409f6d46f7a99, State: Initialized, Role: FOLLOWER
I20260620 14:51:52.196485 15368 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd508df4fdd3436dbab409f6d46f7a99" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34511 } }
I20260620 14:51:52.196578 15368 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:52.196610 15368 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:52.196655 15368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:52.197556 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.195012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56458 (local address 127.5.118.190:34511)
0620 14:51:52.195250 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:52.195260 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:52.195305 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:51:52.196041 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:52.196051 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:52.196104 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:52.196315 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:52.197035 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:52.197040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:52.197040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:52.197045 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:52.197096 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:52.197114 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:52.197198 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:52.197264 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:51:52.197349 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":84,"threads_started":1}
I20260620 14:51:52.197780 15368 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd508df4fdd3436dbab409f6d46f7a99" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34511 } }
I20260620 14:51:52.197894 15368 leader_election.cc:302] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd508df4fdd3436dbab409f6d46f7a99; no voters:
I20260620 14:51:52.198097 15368 leader_election.cc:288] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:52.198130 15373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:52.198235 15373 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [term 1 LEADER]: Becoming Leader. State: Replica: bd508df4fdd3436dbab409f6d46f7a99, State: Running, Role: LEADER
I20260620 14:51:52.198323 15373 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd508df4fdd3436dbab409f6d46f7a99" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34511 } }
I20260620 14:51:52.198391 15368 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:52.198689 15374 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd508df4fdd3436dbab409f6d46f7a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd508df4fdd3436dbab409f6d46f7a99" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34511 } } }
I20260620 14:51:52.198705 15375 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd508df4fdd3436dbab409f6d46f7a99. Latest consensus state: current_term: 1 leader_uuid: "bd508df4fdd3436dbab409f6d46f7a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd508df4fdd3436dbab409f6d46f7a99" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 34511 } } }
I20260620 14:51:52.198817 15374 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:52.198827 15375 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:52.199061 15379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:52.199579 15379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:52.201419 15379 catalog_manager.cc:1347] Generated new cluster ID: a25474280c9b47d9917a41839533a90e
I20260620 14:51:52.201486 15379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:52.219614 15379 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:52.219815 15379 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:52.239941 15379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd508df4fdd3436dbab409f6d46f7a99: Generated new TSK 0
I20260620 14:51:52.240208 15379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:51:52.247284 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34511
--builtin_ntp_servers=127.5.118.148:38307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:51:52.338925 15392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:52.339159 15392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:52.339185 15392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:52.340612 15392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:52.340660 15392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:52.340691 15392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:52.342247 15392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:38307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:52.342535 15392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:52.342895 15392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:52.348243 15398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.348408 15397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.348420 15400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:52.348409 15392 server_base.cc:1034] running on GCE node
I20260620 14:51:52.348728 15392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:52.349015 15392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:52.350186 15392 hybrid_clock.cc:648] HybridClock initialized: now 1781967112350176 us; error 33 us; skew 500 ppm
I20260620 14:51:52.351553 15392 webserver.cc:460] Webserver started at http://127.5.118.129:35549/ using document root <none> and password file <none>
I20260620 14:51:52.351838 15392 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:52.351917 15392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:52.352070 15392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:52.353236 15392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "9cf05de5b124427b961b671b7a9736b3"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "29c95e01868e1e5a28cf394fd714273f"
server_key_iv: "971b86b6040330b5d084ce818d990e3b"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.353609 15392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "9cf05de5b124427b961b671b7a9736b3"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "29c95e01868e1e5a28cf394fd714273f"
server_key_iv: "971b86b6040330b5d084ce818d990e3b"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.354959 15392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:51:52.355703 15406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.355908 15392 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:52.356000 15392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "9cf05de5b124427b961b671b7a9736b3"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "29c95e01868e1e5a28cf394fd714273f"
server_key_iv: "971b86b6040330b5d084ce818d990e3b"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.356098 15392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:52.359323 15392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:52.359617 15392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:52.359761 15392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:52.359982 15392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:52.360345 15392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:52.360400 15392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.360440 15392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:52.360471 15392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.366582 15392 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:40923
I20260620 14:51:52.366660 15519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:40923 every 8 connection(s)
I20260620 14:51:52.367048 15392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:52.370499 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.367943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:41595 (local address 127.5.118.190:34511)
0620 14:51:52.368013 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:52.368014 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:52.368348 (+ 334us) server_negotiation.cc:407] Connection header received
0620 14:51:52.368870 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:52.368872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:52.368879 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:52.368939 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:52.370220 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:52.370222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:52.370222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:52.370227 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:52.370245 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:52.370249 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:52.370316 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:52.370413 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:51:52.370429 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:51:52.370733 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.367929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:34511 (local address 127.5.118.129:41595)
0620 14:51:52.368137 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.368242 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:52.368752 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:52.369016 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:52.369020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:52.369037 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:52.370041 (+ 1004us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:52.370042 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:52.370073 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:52.370115 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:52.370317 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:52.370323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:52.370326 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:52.370427 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:51:52.370491 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260620 14:51:52.371236 15520 heartbeater.cc:347] Connected to a master server at 127.5.118.190:34511
I20260620 14:51:52.371385 15520 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:52.371567 15520 heartbeater.cc:511] Master 127.5.118.190:34511 requested a full tablet report, sending...
I20260620 14:51:52.372159 15332 ts_manager.cc:194] Registered new tserver with Master: 9cf05de5b124427b961b671b7a9736b3 (127.5.118.129:40923)
I20260620 14:51:52.372677 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 15392
I20260620 14:51:52.372797 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:52.373005 5594 external_mini_cluster.cc:1428] Setting key 03e3742baca4347002e51365fd3e0d15
I20260620 14:51:52.374071 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34511
--builtin_ntp_servers=127.5.118.148:38307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:51:52.455411 15523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:52.455685 15523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:52.455708 15523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:52.456947 15523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:52.456992 15523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:52.457019 15523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:52.458468 15523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:38307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:52.458752 15523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:52.459079 15523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:52.464221 15529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.464233 15528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.464241 15531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:52.464385 15523 server_base.cc:1034] running on GCE node
I20260620 14:51:52.464608 15523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:52.464880 15523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:52.466068 15523 hybrid_clock.cc:648] HybridClock initialized: now 1781967112466050 us; error 28 us; skew 500 ppm
I20260620 14:51:52.467430 15523 webserver.cc:460] Webserver started at http://127.5.118.130:33255/ using document root <none> and password file <none>
I20260620 14:51:52.467717 15523 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:52.467765 15523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:52.467912 15523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:52.469058 15523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "c6b1f6b7bf71411682798a2742640dca"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "0e37ae2b683aada267ea183cc2690f45"
server_key_iv: "79d90ae21c5be94902d549b7c88c3f76"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.469417 15523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "c6b1f6b7bf71411682798a2742640dca"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "0e37ae2b683aada267ea183cc2690f45"
server_key_iv: "79d90ae21c5be94902d549b7c88c3f76"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.470813 15523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:51:52.471645 15537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.471832 15523 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:52.471946 15523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "c6b1f6b7bf71411682798a2742640dca"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "0e37ae2b683aada267ea183cc2690f45"
server_key_iv: "79d90ae21c5be94902d549b7c88c3f76"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.472043 15523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:52.475858 15523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:52.476224 15523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:52.476411 15523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:52.476643 15523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:52.476991 15523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:52.477051 15523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.477097 15523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:52.477126 15523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.483771 15523 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:35549
I20260620 14:51:52.483825 15650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:35549 every 8 connection(s)
I20260620 14:51:52.484273 15523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:52.487578 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.485111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:44443 (local address 127.5.118.190:34511)
0620 14:51:52.485181 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:52.485183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:52.485421 (+ 238us) server_negotiation.cc:407] Connection header received
0620 14:51:52.485963 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:52.485964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:52.485972 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:52.486019 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:52.487282 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:52.487293 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:52.487293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:52.487298 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:52.487315 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:52.487320 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:52.487377 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:52.487486 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:51:52.487503 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:51:52.487798 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.485109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:34511 (local address 127.5.118.130:44443)
0620 14:51:52.485305 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.485338 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:52.485842 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:52.486111 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:52.486115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:52.486123 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:52.487117 (+ 994us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:52.487118 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:52.487148 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:52.487179 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:52.487384 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:52.487389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:52.487392 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:52.487480 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:51:52.487547 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260620 14:51:52.488338 15651 heartbeater.cc:347] Connected to a master server at 127.5.118.190:34511
I20260620 14:51:52.488425 15651 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:52.488613 15651 heartbeater.cc:511] Master 127.5.118.190:34511 requested a full tablet report, sending...
I20260620 14:51:52.488780 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 15523
I20260620 14:51:52.488947 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:52.489023 15332 ts_manager.cc:194] Registered new tserver with Master: c6b1f6b7bf71411682798a2742640dca (127.5.118.130:35549)
I20260620 14:51:52.489187 5594 external_mini_cluster.cc:1428] Setting key 241d8401421087884dc03216e843256f
I20260620 14:51:52.490123 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34511
--builtin_ntp_servers=127.5.118.148:38307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:51:52.572767 15654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:52.572975 15654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:52.572997 15654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:52.574199 15654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:52.574242 15654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:52.574270 15654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:52.575726 15654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:38307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:52.576015 15654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:52.576383 15654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:51:52.581607 15654 server_base.cc:1034] running on GCE node
W20260620 14:51:52.581624 15660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.581580 15659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:52.581570 15662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:52.581954 15654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:52.582247 15654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:52.583456 15654 hybrid_clock.cc:648] HybridClock initialized: now 1781967112583425 us; error 45 us; skew 500 ppm
I20260620 14:51:52.584910 15654 webserver.cc:460] Webserver started at http://127.5.118.131:44415/ using document root <none> and password file <none>
I20260620 14:51:52.585125 15654 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:52.585172 15654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:52.585281 15654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:52.586449 15654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "991d10e7483f40f4842ad3d415ceb6d9"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "dd8a819818492d80b07f614400e1dca9"
server_key_iv: "2decdd6dfd844583f35a27df4dc04115"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.586781 15654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "991d10e7483f40f4842ad3d415ceb6d9"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "dd8a819818492d80b07f614400e1dca9"
server_key_iv: "2decdd6dfd844583f35a27df4dc04115"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.588135 15654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:51:52.588969 15668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.589121 15654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:52.589203 15654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "991d10e7483f40f4842ad3d415ceb6d9"
format_stamp: "Formatted at 2026-06-20 14:51:52 on dist-test-slave-2m5f"
server_key: "dd8a819818492d80b07f614400e1dca9"
server_key_iv: "2decdd6dfd844583f35a27df4dc04115"
server_key_version: "encryptionkey@0"
I20260620 14:51:52.589258 15654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:52.592617 15654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:52.592840 15654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:52.592932 15654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:52.593075 15654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:52.593367 15654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:52.593398 15654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.593418 15654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:52.593461 15654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:52.599128 15654 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:39601
I20260620 14:51:52.599236 15781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:39601 every 8 connection(s)
I20260620 14:51:52.599542 15654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:52.602999 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.600420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:43129 (local address 127.5.118.190:34511)
0620 14:51:52.600497 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:52.600499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:52.600810 (+ 311us) server_negotiation.cc:407] Connection header received
0620 14:51:52.601419 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:52.601420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:52.601427 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:52.601476 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:52.602730 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:52.602732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:52.602732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:51:52.602737 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:52.602753 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:51:52.602758 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:52.602812 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:52.602910 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:51:52.602925 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:51:52.603242 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.600440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:34511 (local address 127.5.118.131:43129)
0620 14:51:52.600669 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.600704 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:52.601300 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:52.601554 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:52.601558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:52.601566 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:52.602556 (+ 990us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:51:52.602556 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:52.602585 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:51:52.602618 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:52.602816 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:52.602820 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:52.602823 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:52.602921 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:51:52.602985 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260620 14:51:52.603714 15782 heartbeater.cc:347] Connected to a master server at 127.5.118.190:34511
I20260620 14:51:52.603798 15782 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:52.603957 15782 heartbeater.cc:511] Master 127.5.118.190:34511 requested a full tablet report, sending...
I20260620 14:51:52.604374 15332 ts_manager.cc:194] Registered new tserver with Master: 991d10e7483f40f4842ad3d415ceb6d9 (127.5.118.131:39601)
I20260620 14:51:52.604998 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 15654
I20260620 14:51:52.605070 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:52.605235 5594 external_mini_cluster.cc:1428] Setting key f7a0abb2326307aa9a554b6e2acbf683
I20260620 14:51:52.606197 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:51:52.607906 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 15392
W20260620 14:51:52.607905 15372 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:51:52.607394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.5.118.190:34511)
0620 14:51:52.607481 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:52.607483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:52.607580 (+ 97us) server_negotiation.cc:407] Connection header received
0620 14:51:52.607606 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:52.607607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:52.607612 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:52.607664 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:52.607815 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56466: BlockingRecv error: recv got EOF from 127.0.0.1:56466 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:51:52.613328 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 15523
I20260620 14:51:52.617763 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 15654
I20260620 14:51:52.622026 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 15301
2026-06-20T14:51:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (579 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:52 dist-test-slave-2m5f krb5kdc[15792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:52 dist-test-slave-2m5f krb5kdc[15792](info): set up 2 sockets
Jun 20 14:51:52 dist-test-slave-2m5f krb5kdc[15792](info): commencing operation
krb5kdc: starting...
W20260620 14:51:53.681074 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:53 dist-test-slave-2m5f krb5kdc[15792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:51:53.800689 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:36609
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:36609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:53.885546 15805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:53.885811 15805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:53.885852 15805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:53.887105 15805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:53.887180 15805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:53.887204 15805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:53.887226 15805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:53.887264 15805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:53.888834 15805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:36609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:36609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:53.889132 15805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:53.889374 15805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:53.894644 15813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:53.894644 15811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:53.894690 15810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:53.894924 15805 server_base.cc:1034] running on GCE node
I20260620 14:51:53.895171 15805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:53.895459 15805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:53.896626 15805 hybrid_clock.cc:648] HybridClock initialized: now 1781967113896620 us; error 38 us; skew 500 ppm
Jun 20 14:51:53 dist-test-slave-2m5f krb5kdc[15792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967113, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:53.899026 15805 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:51:53.899546 15805 webserver.cc:460] Webserver started at http://127.5.118.190:44619/ using document root <none> and password file <none>
I20260620 14:51:53.899736 15805 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:53.899783 15805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:53.899873 15805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:53.900971 15805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "25e5715bbcac44bda15e6fc9bdee610f"
format_stamp: "Formatted at 2026-06-20 14:51:53 on dist-test-slave-2m5f"
server_key: "ddd24d03d0bdb5788d6d225e25dc0891"
server_key_iv: "488922631ecdb651b804405172313615"
server_key_version: "encryptionkey@0"
I20260620 14:51:53.901288 15805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "25e5715bbcac44bda15e6fc9bdee610f"
format_stamp: "Formatted at 2026-06-20 14:51:53 on dist-test-slave-2m5f"
server_key: "ddd24d03d0bdb5788d6d225e25dc0891"
server_key_iv: "488922631ecdb651b804405172313615"
server_key_version: "encryptionkey@0"
I20260620 14:51:53.902554 15805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:51:53.903296 15820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:53.903517 15805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:51:53.903643 15805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "25e5715bbcac44bda15e6fc9bdee610f"
format_stamp: "Formatted at 2026-06-20 14:51:53 on dist-test-slave-2m5f"
server_key: "ddd24d03d0bdb5788d6d225e25dc0891"
server_key_iv: "488922631ecdb651b804405172313615"
server_key_version: "encryptionkey@0"
I20260620 14:51:53.903710 15805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:53.933362 15805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:53.935788 15805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:53.935925 15805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:53.939920 15805 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:36609
I20260620 14:51:53.940014 15872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:36609 every 8 connection(s)
I20260620 14:51:53.940408 15805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:53.941479 15873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:53.943938 15873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f: Bootstrap starting.
I20260620 14:51:53.944584 15873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:53.944881 15873 log.cc:824] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:53.945684 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 15805
I20260620 14:51:53.945755 15873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f: No bootstrap required, opened a new log
I20260620 14:51:53.945787 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:51:53.946003 5594 external_mini_cluster.cc:1428] Setting key f7f86729fa979f52a74708740ff622bb
I20260620 14:51:53.947649 15873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5715bbcac44bda15e6fc9bdee610f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36609 } }
I20260620 14:51:53.947795 15873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:53.947817 15873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e5715bbcac44bda15e6fc9bdee610f, State: Initialized, Role: FOLLOWER
I20260620 14:51:53.947970 15873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5715bbcac44bda15e6fc9bdee610f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36609 } }
I20260620 14:51:53.948069 15873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:53.948107 15873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:53.948177 15873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:53.949080 15873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5715bbcac44bda15e6fc9bdee610f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36609 } }
I20260620 14:51:53.949205 15873 leader_election.cc:302] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25e5715bbcac44bda15e6fc9bdee610f; no voters:
I20260620 14:51:53.949395 15873 leader_election.cc:288] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:53.949417 15878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:53.949546 15878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [term 1 LEADER]: Becoming Leader. State: Replica: 25e5715bbcac44bda15e6fc9bdee610f, State: Running, Role: LEADER
I20260620 14:51:53.949723 15873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:53.949695 15878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5715bbcac44bda15e6fc9bdee610f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36609 } }
I20260620 14:51:53.951603 15880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25e5715bbcac44bda15e6fc9bdee610f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5715bbcac44bda15e6fc9bdee610f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36609 } } }
W20260620 14:51:53.951663 15893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 14:51:53.951603 15879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e5715bbcac44bda15e6fc9bdee610f. Latest consensus state: current_term: 1 leader_uuid: "25e5715bbcac44bda15e6fc9bdee610f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e5715bbcac44bda15e6fc9bdee610f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 36609 } } }
W20260620 14:51:53.951864 15893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:51:53.951887 15879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:53.951785 15880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:53.952108 15895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:53.952316 15895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:53.954523 15895 catalog_manager.cc:1347] Generated new cluster ID: 3839bb5452734a11ac4302449bd687fd
I20260620 14:51:53.954577 15895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:51:53 dist-test-slave-2m5f krb5kdc[15792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:53.959474 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:53.946573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.5.118.190:36609)
0620 14:51:53.946812 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:53.946821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:53.946848 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:51:53.947651 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:53.947659 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:53.947734 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:53.947932 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:53.948715 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:53.949992 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:53.951217 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:53.951947 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:53.952073 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:53.955297 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:53.955301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:53.955302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:53.955323 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:53.957584 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:53.958024 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:53.958028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:53.958028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:53.958180 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:53.958398 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:53.958400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:53.958400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:53.958770 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:53.958965 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:53.959042 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:51:53.959212 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260620 14:51:53.962261 15895 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:53.963001 15895 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:53.968235 15895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f: Generated new TSK 0
I20260620 14:51:53.968438 15895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:51:53.999662 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36609
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:54.083006 15899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:54.083220 15899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:54.083245 15899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:54.084553 15899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:54.084595 15899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:54.084623 15899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:51:54.086133 15899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:54.086395 15899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:54.086658 15899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:54.092489 15904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:54.092521 15907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:54.092504 15905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:54.092690 15899 server_base.cc:1034] running on GCE node
I20260620 14:51:54.093010 15899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:54.093218 15899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:54.094384 15899 hybrid_clock.cc:648] HybridClock initialized: now 1781967114094382 us; error 36 us; skew 500 ppm
Jun 20 14:51:54 dist-test-slave-2m5f krb5kdc[15792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:54.096994 15899 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:51:54.097431 15899 webserver.cc:460] Webserver started at http://127.5.118.129:44263/ using document root <none> and password file <none>
I20260620 14:51:54.097684 15899 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:54.097759 15899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:54.097908 15899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:54.098973 15899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "eb7563e30f0e4094a97634dea86e4929"
format_stamp: "Formatted at 2026-06-20 14:51:54 on dist-test-slave-2m5f"
server_key: "4d0937dba51a886ce5b276dda1eac73a"
server_key_iv: "29c27d4db97b602e79548ff32ce440ee"
server_key_version: "encryptionkey@0"
I20260620 14:51:54.099380 15899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "eb7563e30f0e4094a97634dea86e4929"
format_stamp: "Formatted at 2026-06-20 14:51:54 on dist-test-slave-2m5f"
server_key: "4d0937dba51a886ce5b276dda1eac73a"
server_key_iv: "29c27d4db97b602e79548ff32ce440ee"
server_key_version: "encryptionkey@0"
I20260620 14:51:54.100832 15899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:51:54.101635 15914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:54.101815 15899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:51:54.101923 15899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "eb7563e30f0e4094a97634dea86e4929"
format_stamp: "Formatted at 2026-06-20 14:51:54 on dist-test-slave-2m5f"
server_key: "4d0937dba51a886ce5b276dda1eac73a"
server_key_iv: "29c27d4db97b602e79548ff32ce440ee"
server_key_version: "encryptionkey@0"
I20260620 14:51:54.102001 15899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:54.112848 15899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:54.114740 15899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:54.114915 15899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:54.115154 15899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:54.115480 15899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:54.115537 15899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:54.115574 15899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:54.115602 15899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:54.121554 15899 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:39067
I20260620 14:51:54.121593 16027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:39067 every 8 connection(s)
I20260620 14:51:54.121970 15899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:54.124760 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 15899
I20260620 14:51:54.124871 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:54.125052 5594 external_mini_cluster.cc:1428] Setting key 67231df18f30a246cf985cf78bc0ed10
Jun 20 14:51:54 dist-test-slave-2m5f krb5kdc[15792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:54.134611 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.122733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:47453 (local address 127.5.118.190:36609)
0620 14:51:54.122782 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.122784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.122973 (+ 189us) server_negotiation.cc:407] Connection header received
0620 14:51:54.123553 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.123554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.123567 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.123620 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:54.124947 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.125775 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.127384 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.127740 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.127820 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.130239 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:54.130241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:54.130241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:54.130274 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:54.132304 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.133110 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.133112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.133112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.133181 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.133630 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.133631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.133631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.133792 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:54.133859 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.134393 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.134509 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
I20260620 14:51:54.135067 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.122693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:36609 (local address 127.5.118.129:47453)
0620 14:51:54.122832 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.122881 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:54.123436 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:54.123706 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:54.123711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:54.124392 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:54.124685 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.124693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.125891 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.125892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.127281 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.127284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.127834 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.127835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.127947 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.128578 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:54.128589 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:54.130119 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:54.132426 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:54.132435 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:54.132438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:54.133009 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:54.133294 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:54.133295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:54.133296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:54.133542 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:54.133864 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:54.133870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:54.134177 (+ 307us) client_negotiation.cc:769] Sending connection context
0620 14:51:54.134365 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 14:51:54.134646 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260620 14:51:54.135967 16028 heartbeater.cc:347] Connected to a master server at 127.5.118.190:36609
I20260620 14:51:54.136085 16028 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:54.136323 16028 heartbeater.cc:511] Master 127.5.118.190:36609 requested a full tablet report, sending...
I20260620 14:51:54.137046 15837 ts_manager.cc:194] Registered new tserver with Master: eb7563e30f0e4094a97634dea86e4929 (127.5.118.129:39067)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
I20260620 14:51:54.138083 15837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:47453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:51:54.154844 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36609
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:54.237093 16033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:54.237329 16033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:54.237355 16033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:54.238600 16033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:54.238648 16033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:54.238677 16033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:51:54.240239 16033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:54.240509 16033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:54.240752 16033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:54.245868 16038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:54.246006 16033 server_base.cc:1034] running on GCE node
W20260620 14:51:54.245850 16041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:54.245850 16039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:54.246289 16033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:54.246546 16033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:54.247713 16033 hybrid_clock.cc:648] HybridClock initialized: now 1781967114247691 us; error 54 us; skew 500 ppm
Jun 20 14:51:54 dist-test-slave-2m5f krb5kdc[15792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:54.250507 16033 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:51:54.250989 16033 webserver.cc:460] Webserver started at http://127.5.118.130:40799/ using document root <none> and password file <none>
I20260620 14:51:54.251230 16033 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:54.251300 16033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:54.251453 16033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:54.252622 16033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "026ee74dcbfd468fa00b5696bd3dfac9"
format_stamp: "Formatted at 2026-06-20 14:51:54 on dist-test-slave-2m5f"
server_key: "1612e320e3fbfb35e30cc00913d0722e"
server_key_iv: "5d43f247a73b5b2d9576cae37cac56a1"
server_key_version: "encryptionkey@0"
I20260620 14:51:54.252995 16033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "026ee74dcbfd468fa00b5696bd3dfac9"
format_stamp: "Formatted at 2026-06-20 14:51:54 on dist-test-slave-2m5f"
server_key: "1612e320e3fbfb35e30cc00913d0722e"
server_key_iv: "5d43f247a73b5b2d9576cae37cac56a1"
server_key_version: "encryptionkey@0"
I20260620 14:51:54.254359 16033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:51:54.255175 16048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:54.255369 16033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:51:54.255475 16033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "026ee74dcbfd468fa00b5696bd3dfac9"
format_stamp: "Formatted at 2026-06-20 14:51:54 on dist-test-slave-2m5f"
server_key: "1612e320e3fbfb35e30cc00913d0722e"
server_key_iv: "5d43f247a73b5b2d9576cae37cac56a1"
server_key_version: "encryptionkey@0"
I20260620 14:51:54.255554 16033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:54.297465 16033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:54.299798 16033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:54.299988 16033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:54.300333 16033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:54.300686 16033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:54.300743 16033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:54.300787 16033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:54.300813 16033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:54.306912 16033 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:35147
I20260620 14:51:54.306965 16161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:35147 every 8 connection(s)
I20260620 14:51:54.307356 16033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:54.310438 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 16033
I20260620 14:51:54.310528 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:54.310732 5594 external_mini_cluster.cc:1428] Setting key 3c38c90ac9d1d11fc926ea2339fa5804
Jun 20 14:51:54 dist-test-slave-2m5f krb5kdc[15792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:54.318239 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.308188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:41091 (local address 127.5.118.190:36609)
0620 14:51:54.308259 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.308261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.308451 (+ 190us) server_negotiation.cc:407] Connection header received
0620 14:51:54.308947 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.308948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.308958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.309023 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:54.309964 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.310530 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.311822 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.312034 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.312114 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.314936 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:54.314937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:54.314938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:54.314944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:54.316498 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.317048 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.317048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.317049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.317100 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.317435 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.317436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.317437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.317565 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:54.317662 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.318070 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.318164 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:51:54.318516 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.308130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:36609 (local address 127.5.118.130:41091)
0620 14:51:54.308317 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.308361 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:54.308860 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:54.309079 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:54.309083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:54.309671 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:54.309876 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.309880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.310644 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.310645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.311699 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.311701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.312120 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.312122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.312285 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.312851 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:54.312863 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:54.314840 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:54.316583 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:54.316592 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:54.316595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:54.316966 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:54.317170 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:54.317171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:54.317171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:54.317335 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:54.317647 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:54.317655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:54.317909 (+ 254us) client_negotiation.cc:769] Sending connection context
0620 14:51:54.318037 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:51:54.318234 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":118,"threads_started":1}
I20260620 14:51:54.319057 16162 heartbeater.cc:347] Connected to a master server at 127.5.118.190:36609
I20260620 14:51:54.319255 16162 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:54.319460 16162 heartbeater.cc:511] Master 127.5.118.190:36609 requested a full tablet report, sending...
I20260620 14:51:54.319909 15837 ts_manager.cc:194] Registered new tserver with Master: 026ee74dcbfd468fa00b5696bd3dfac9 (127.5.118.130:35147)
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
I20260620 14:51:54.320499 15837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:41091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:51:54.338440 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36609
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:42031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:54.420511 16167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:54.420711 16167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:54.420734 16167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:54.421926 16167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:54.421968 16167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:54.421995 16167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:51:54.423449 16167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:36609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:51:54.423700 16167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:54.423960 16167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:54.428969 16175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:54.429023 16173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:54.428957 16172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:54.429195 16167 server_base.cc:1034] running on GCE node
I20260620 14:51:54.429481 16167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:54.429683 16167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:54.430819 16167 hybrid_clock.cc:648] HybridClock initialized: now 1781967114430808 us; error 28 us; skew 500 ppm
Jun 20 14:51:54 dist-test-slave-2m5f krb5kdc[15792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:54.433461 16167 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:51:54.433928 16167 webserver.cc:460] Webserver started at http://127.5.118.131:42493/ using document root <none> and password file <none>
I20260620 14:51:54.434129 16167 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:54.434170 16167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:54.434267 16167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:54.435261 16167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "67cd19de53d94e0fac47e9f0911e5ddd"
format_stamp: "Formatted at 2026-06-20 14:51:54 on dist-test-slave-2m5f"
server_key: "8c0a953d9076faee1dd8d183700bfc8a"
server_key_iv: "2c43ee47afee5c29eddcff99118acf4d"
server_key_version: "encryptionkey@0"
I20260620 14:51:54.435565 16167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "67cd19de53d94e0fac47e9f0911e5ddd"
format_stamp: "Formatted at 2026-06-20 14:51:54 on dist-test-slave-2m5f"
server_key: "8c0a953d9076faee1dd8d183700bfc8a"
server_key_iv: "2c43ee47afee5c29eddcff99118acf4d"
server_key_version: "encryptionkey@0"
I20260620 14:51:54.436955 16167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:51:54.437692 16182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:54.437891 16167 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:51:54.437981 16167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "67cd19de53d94e0fac47e9f0911e5ddd"
format_stamp: "Formatted at 2026-06-20 14:51:54 on dist-test-slave-2m5f"
server_key: "8c0a953d9076faee1dd8d183700bfc8a"
server_key_iv: "2c43ee47afee5c29eddcff99118acf4d"
server_key_version: "encryptionkey@0"
I20260620 14:51:54.438057 16167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:54.454349 16167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:54.456756 16167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:54.456911 16167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:54.457180 16167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:54.457513 16167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:54.457554 16167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:54.457578 16167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:54.457592 16167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:54.463770 16167 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:36755
I20260620 14:51:54.463816 16295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:36755 every 8 connection(s)
I20260620 14:51:54.464135 16167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:51:54 dist-test-slave-2m5f krb5kdc[15792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:51:54.473335 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.464955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:41337 (local address 127.5.118.190:36609)
0620 14:51:54.465002 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.465004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.465205 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:51:54.465671 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.465672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.465682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.465743 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:54.466673 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.467171 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.468332 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.468583 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.468641 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.470829 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:54.470830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:54.470831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:54.470837 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:54.472073 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.472493 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.472494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.472494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.472547 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.472790 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.472791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.472791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.472898 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:54.472964 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.473195 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.473258 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:51:54.473536 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.464906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:36609 (local address 127.5.118.131:41337)
0620 14:51:54.465066 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.465108 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:54.465588 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:54.465801 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:54.465804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:54.466383 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:54.466581 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.466586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.467288 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.467289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.468089 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.468090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.468673 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.468674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.468790 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.469358 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:54.469367 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:54.470735 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:54.472147 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:54.472153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:54.472155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:54.472428 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:54.472615 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:54.472615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:54.472616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:54.472717 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:54.472961 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:54.472965 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:54.473083 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 14:51:54.473167 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:51:54.473320 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260620 14:51:54.473732 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 16167
I20260620 14:51:54.473846 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:54.474012 16296 heartbeater.cc:347] Connected to a master server at 127.5.118.190:36609
I20260620 14:51:54.474080 5594 external_mini_cluster.cc:1428] Setting key a620bf17ba5cd0c437f2fba95a21d6a0
I20260620 14:51:54.474113 16296 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:54.474335 16296 heartbeater.cc:511] Master 127.5.118.190:36609 requested a full tablet report, sending...
I20260620 14:51:54.474740 15836 ts_manager.cc:194] Registered new tserver with Master: 67cd19de53d94e0fac47e9f0911e5ddd (127.5.118.131:36755)
I20260620 14:51:54.475159 15836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:41337
I20260620 14:51:54.485795 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:51:54.493397 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.487121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.5.118.190:36609)
0620 14:51:54.487195 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.487197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.487324 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:51:54.487405 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.487405 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.487416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.487466 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:54.488035 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.488601 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.489333 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.489541 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.489627 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.490934 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:54.490935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:54.490936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:54.490944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:54.492228 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.492564 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.492565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.492566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.492628 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:54.492832 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:54.492833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:54.492833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:54.492990 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:54.493094 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.493214 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.493292 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:51:54.494968 15836 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:54.496724 15836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:54.503114 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.499232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.5.118.130:35147)
0620 14:51:54.499419 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.499423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.499459 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:51:54.499531 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.499534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.499549 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.499617 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:54.499847 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.500957 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.502266 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.502739 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.502843 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.502905 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.502997 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.503032 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260620 14:51:54.503118 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.498987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:36755 (local address 127.0.0.1:40232)
0620 14:51:54.499146 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.499184 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:54.499249 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:54.499616 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:54.499636 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:54.499638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:54.499708 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.499710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.501080 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.501081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.501929 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.501931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.502819 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.502820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.502901 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.502916 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:51:54.502963 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:51:54.503014 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260620 14:51:54.503274 16308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.498991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39067 (local address 127.0.0.1:45050)
0620 14:51:54.499536 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.499543 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:54.499600 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:54.499877 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:54.499878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:54.499880 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:54.499918 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.499919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.500870 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.500871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.502146 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.502147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.503059 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.503060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.503112 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.503115 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:51:54.503162 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:51:54.503196 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260620 14:51:54.503273 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.499065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.5.118.131:36755)
0620 14:51:54.499443 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.499446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.499456 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:51:54.499494 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.499496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.499513 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.499595 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:54.499804 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.500949 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.502035 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.502705 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.503077 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.503133 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.503156 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.503195 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":71,"threads_started":1}
I20260620 14:51:54.503476 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.499220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.5.118.129:39067)
0620 14:51:54.499386 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.499393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.499654 (+ 261us) server_negotiation.cc:407] Connection header received
0620 14:51:54.499698 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.499700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.499715 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.499789 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:54.499992 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.500768 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.502225 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.502915 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.502978 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.503027 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.503374 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.503418 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260620 14:51:54.503546 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.498989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:35147 (local address 127.0.0.1:34202)
0620 14:51:54.499400 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.499410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:54.499475 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:54.499687 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:54.499688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:54.499691 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:54.499740 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.499741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.501058 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.501058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.502134 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.502136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.502847 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.502848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.502927 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.502932 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:51:54.502986 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:51:54.503029 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":81,"threads_started":1}
I20260620 14:51:54.503973 16230 tablet_service.cc:1467] Processing CreateTablet for tablet 32be767b586348c48a049fc5bef277cf (DEFAULT_TABLE table=test-table [id=7024dbc3ae9c4aee81f98ba09ed52cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:54.504379 16230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32be767b586348c48a049fc5bef277cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:54.504274 15962 tablet_service.cc:1467] Processing CreateTablet for tablet 32be767b586348c48a049fc5bef277cf (DEFAULT_TABLE table=test-table [id=7024dbc3ae9c4aee81f98ba09ed52cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:54.504583 15962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32be767b586348c48a049fc5bef277cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:54.504613 16096 tablet_service.cc:1467] Processing CreateTablet for tablet 32be767b586348c48a049fc5bef277cf (DEFAULT_TABLE table=test-table [id=7024dbc3ae9c4aee81f98ba09ed52cb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:54.504985 16096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32be767b586348c48a049fc5bef277cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:54.507841 16313 tablet_bootstrap.cc:492] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: Bootstrap starting.
I20260620 14:51:54.508373 16314 tablet_bootstrap.cc:492] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929: Bootstrap starting.
I20260620 14:51:54.508525 16313 tablet_bootstrap.cc:654] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:54.508939 16314 tablet_bootstrap.cc:654] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:54.509054 16313 log.cc:824] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:54.509454 16314 log.cc:824] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:54.509769 16315 tablet_bootstrap.cc:492] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: Bootstrap starting.
I20260620 14:51:54.509836 16313 tablet_bootstrap.cc:492] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: No bootstrap required, opened a new log
I20260620 14:51:54.509913 16313 ts_tablet_manager.cc:1397] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:51:54.510277 16314 tablet_bootstrap.cc:492] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929: No bootstrap required, opened a new log
I20260620 14:51:54.510350 16314 ts_tablet_manager.cc:1397] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:51:54.510422 16315 tablet_bootstrap.cc:654] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:54.511011 16315 log.cc:824] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:54.511663 16315 tablet_bootstrap.cc:492] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: No bootstrap required, opened a new log
I20260620 14:51:54.511742 16315 ts_tablet_manager.cc:1397] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:51:54.511940 16313 raft_consensus.cc:348] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36755 } } peers { permanent_uuid: "eb7563e30f0e4094a97634dea86e4929" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39067 } } peers { permanent_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35147 } }
I20260620 14:51:54.512122 16313 raft_consensus.cc:374] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:54.512161 16313 raft_consensus.cc:729] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026ee74dcbfd468fa00b5696bd3dfac9, State: Initialized, Role: FOLLOWER
I20260620 14:51:54.512341 16313 consensus_queue.cc:260] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36755 } } peers { permanent_uuid: "eb7563e30f0e4094a97634dea86e4929" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39067 } } peers { permanent_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35147 } }
I20260620 14:51:54.512280 16314 raft_consensus.cc:348] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36755 } } peers { permanent_uuid: "eb7563e30f0e4094a97634dea86e4929" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39067 } } peers { permanent_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35147 } }
I20260620 14:51:54.512434 16314 raft_consensus.cc:374] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:54.512481 16314 raft_consensus.cc:729] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7563e30f0e4094a97634dea86e4929, State: Initialized, Role: FOLLOWER
I20260620 14:51:54.512631 16313 ts_tablet_manager.cc:1428] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:51:54.512631 16314 consensus_queue.cc:260] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36755 } } peers { permanent_uuid: "eb7563e30f0e4094a97634dea86e4929" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39067 } } peers { permanent_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35147 } }
I20260620 14:51:54.512758 16162 heartbeater.cc:503] Master 127.5.118.190:36609 was elected leader, sending a full tablet report...
I20260620 14:51:54.512909 16314 ts_tablet_manager.cc:1428] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:54.513010 16028 heartbeater.cc:503] Master 127.5.118.190:36609 was elected leader, sending a full tablet report...
I20260620 14:51:54.514307 16315 raft_consensus.cc:348] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36755 } } peers { permanent_uuid: "eb7563e30f0e4094a97634dea86e4929" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39067 } } peers { permanent_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35147 } }
I20260620 14:51:54.514464 16315 raft_consensus.cc:374] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:54.514508 16315 raft_consensus.cc:729] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67cd19de53d94e0fac47e9f0911e5ddd, State: Initialized, Role: FOLLOWER
I20260620 14:51:54.514621 16315 consensus_queue.cc:260] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36755 } } peers { permanent_uuid: "eb7563e30f0e4094a97634dea86e4929" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39067 } } peers { permanent_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35147 } }
I20260620 14:51:54.514874 16315 ts_tablet_manager.cc:1428] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:54.514935 16296 heartbeater.cc:503] Master 127.5.118.190:36609 was elected leader, sending a full tablet report...
W20260620 14:51:54.557951 16163 tablet.cc:2369] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:51:54.622781 16029 tablet.cc:2369] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:51:54.714746 16297 tablet.cc:2369] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:54.760380 16321 raft_consensus.cc:482] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:54.760502 16321 raft_consensus.cc:504] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36755 } } peers { permanent_uuid: "eb7563e30f0e4094a97634dea86e4929" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39067 } } peers { permanent_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35147 } }
I20260620 14:51:54.760864 16321 leader_election.cc:288] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb7563e30f0e4094a97634dea86e4929 (127.5.118.129:39067), 026ee74dcbfd468fa00b5696bd3dfac9 (127.5.118.130:35147)
I20260620 14:51:54.763937 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.761115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:49899 (local address 127.5.118.129:39067)
0620 14:51:54.761159 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.761161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.761243 (+ 82us) server_negotiation.cc:407] Connection header received
0620 14:51:54.761281 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.761283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.761295 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.761361 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:54.761551 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.762154 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.763149 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.763558 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.763615 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.763622 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.763814 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.763863 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:51:54.763988 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.761208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:48725 (local address 127.5.118.130:35147)
0620 14:51:54.761303 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.761305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.761590 (+ 285us) server_negotiation.cc:407] Connection header received
0620 14:51:54.761622 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.761623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.761631 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.761668 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:54.761812 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.762326 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.763248 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.763631 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.763705 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.763712 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.763886 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.763928 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:51:54.763981 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.761085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39067 (local address 127.5.118.131:49899)
0620 14:51:54.761144 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.761157 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:54.761215 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:54.761381 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:54.761382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:54.761384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:54.761457 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.761459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.762228 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.762229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.763049 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.763051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.763635 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.763636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.763723 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.763727 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:51:54.763796 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:51:54.763829 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 14:51:54.764003 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.761090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:35147 (local address 127.5.118.131:48725)
0620 14:51:54.761486 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.761495 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:54.761576 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:54.761679 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:54.761679 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:54.761681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:54.761748 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.761749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.762401 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.762402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.763159 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:54.763161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.763714 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.763714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:54.763798 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.763803 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:51:54.763863 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:51:54.763898 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":207,"thread_start_us":125,"threads_started":1}
I20260620 14:51:54.764362 15982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32be767b586348c48a049fc5bef277cf" candidate_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7563e30f0e4094a97634dea86e4929" is_pre_election: true
I20260620 14:51:54.764494 15982 raft_consensus.cc:2454] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67cd19de53d94e0fac47e9f0911e5ddd in term 0.
I20260620 14:51:54.764588 16116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32be767b586348c48a049fc5bef277cf" candidate_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" is_pre_election: true
I20260620 14:51:54.764711 16116 raft_consensus.cc:2454] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67cd19de53d94e0fac47e9f0911e5ddd in term 0.
I20260620 14:51:54.764730 16186 leader_election.cc:302] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67cd19de53d94e0fac47e9f0911e5ddd, eb7563e30f0e4094a97634dea86e4929; no voters:
I20260620 14:51:54.764851 16321 raft_consensus.cc:2781] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:54.764928 16321 raft_consensus.cc:482] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:54.764964 16321 raft_consensus.cc:3037] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:54.765707 16321 raft_consensus.cc:504] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36755 } } peers { permanent_uuid: "eb7563e30f0e4094a97634dea86e4929" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39067 } } peers { permanent_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35147 } }
I20260620 14:51:54.765867 16321 leader_election.cc:288] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [CANDIDATE]: Term 1 election: Requested vote from peers eb7563e30f0e4094a97634dea86e4929 (127.5.118.129:39067), 026ee74dcbfd468fa00b5696bd3dfac9 (127.5.118.130:35147)
I20260620 14:51:54.766013 15982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32be767b586348c48a049fc5bef277cf" candidate_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7563e30f0e4094a97634dea86e4929"
I20260620 14:51:54.766057 16116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32be767b586348c48a049fc5bef277cf" candidate_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026ee74dcbfd468fa00b5696bd3dfac9"
I20260620 14:51:54.766098 15982 raft_consensus.cc:3037] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:54.766153 16116 raft_consensus.cc:3037] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:54.766819 15982 raft_consensus.cc:2454] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67cd19de53d94e0fac47e9f0911e5ddd in term 1.
I20260620 14:51:54.766955 16116 raft_consensus.cc:2454] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67cd19de53d94e0fac47e9f0911e5ddd in term 1.
I20260620 14:51:54.767014 16186 leader_election.cc:302] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67cd19de53d94e0fac47e9f0911e5ddd, eb7563e30f0e4094a97634dea86e4929; no voters:
I20260620 14:51:54.767099 16321 raft_consensus.cc:2781] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:54.767251 16321 raft_consensus.cc:686] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 1 LEADER]: Becoming Leader. State: Replica: 67cd19de53d94e0fac47e9f0911e5ddd, State: Running, Role: LEADER
I20260620 14:51:54.767366 16321 consensus_queue.cc:237] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36755 } } peers { permanent_uuid: "eb7563e30f0e4094a97634dea86e4929" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39067 } } peers { permanent_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35147 } }
I20260620 14:51:54.768302 15837 catalog_manager.cc:5495] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 67cd19de53d94e0fac47e9f0911e5ddd (127.5.118.131). New cstate: current_term: 1 leader_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67cd19de53d94e0fac47e9f0911e5ddd" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb7563e30f0e4094a97634dea86e4929" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35147 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:54.788779 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.786453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.5.118.131:36755)
0620 14:51:54.786503 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:54.786505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:54.786513 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:51:54.786592 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:54.786593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:54.786610 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:54.786672 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:54.786783 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.787259 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.788031 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:54.788268 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:54.788323 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:54.788443 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:54.788542 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:54.788596 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:54.788625 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 14:51:54.788641 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 14:51:54.790613 15982 raft_consensus.cc:1261] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929 [term 1 FOLLOWER]: Refusing update from remote peer 67cd19de53d94e0fac47e9f0911e5ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:54.790623 16116 raft_consensus.cc:1261] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9 [term 1 FOLLOWER]: Refusing update from remote peer 67cd19de53d94e0fac47e9f0911e5ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:54.790988 16321 consensus_queue.cc:1035] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb7563e30f0e4094a97634dea86e4929" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:54.791168 16323 consensus_queue.cc:1035] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "026ee74dcbfd468fa00b5696bd3dfac9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:54.793483 16326 mvcc.cc:205] Tried to move back new op lower bound from 7298937302179545088 to 7298937302087745536. Current Snapshot: MvccSnapshot[applied={T|T < 7298937302179545088}]
I20260620 14:51:54.793612 16327 mvcc.cc:205] Tried to move back new op lower bound from 7298937302179545088 to 7298937302087745536. Current Snapshot: MvccSnapshot[applied={T|T < 7298937302179545088}]
I20260620 14:51:54.796562 15837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:54.796656 15837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:54.797433 15837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25e5715bbcac44bda15e6fc9bdee610f: Sending DeleteTablet for 3 replicas of tablet 32be767b586348c48a049fc5bef277cf
I20260620 14:51:54.797736 16230 tablet_service.cc:1514] Processing DeleteTablet for tablet 32be767b586348c48a049fc5bef277cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:54 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:40232
I20260620 14:51:54.797804 16096 tablet_service.cc:1514] Processing DeleteTablet for tablet 32be767b586348c48a049fc5bef277cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:54 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:34202
I20260620 14:51:54.798017 15962 tablet_service.cc:1514] Processing DeleteTablet for tablet 32be767b586348c48a049fc5bef277cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:54 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:45050
I20260620 14:51:54.798151 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 15899
I20260620 14:51:54.798234 16337 tablet_replica.cc:330] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: stopping tablet replica
I20260620 14:51:54.798262 16338 tablet_replica.cc:330] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929: stopping tablet replica
I20260620 14:51:54.798348 16337 raft_consensus.cc:2229] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:54.798454 16337 raft_consensus.cc:2258] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:54.798457 16338 raft_consensus.cc:2229] T 32be767b586348c48a049fc5bef277cf P eb7563e30f0e4094a97634dea86e4929 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:54.798938 16337 ts_tablet_manager.cc:1905] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:54.800949 16337 ts_tablet_manager.cc:1918] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:54.801031 16337 log.cc:1196] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/32be767b586348c48a049fc5bef277cf
I20260620 14:51:54.801302 16337 ts_tablet_manager.cc:1939] T 32be767b586348c48a049fc5bef277cf P 026ee74dcbfd468fa00b5696bd3dfac9: Deleting consensus metadata
I20260620 14:51:54.801730 16336 tablet_replica.cc:330] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: stopping tablet replica
I20260620 14:51:54.801781 15822 catalog_manager.cc:4841] TS 026ee74dcbfd468fa00b5696bd3dfac9 (127.5.118.130:35147): tablet 32be767b586348c48a049fc5bef277cf (table test-table [id=7024dbc3ae9c4aee81f98ba09ed52cb4]) successfully deleted
I20260620 14:51:54.801841 16336 raft_consensus.cc:2229] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:54.801970 16336 raft_consensus.cc:2258] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:54.802433 16336 ts_tablet_manager.cc:1905] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:54.804430 16336 ts_tablet_manager.cc:1918] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:54.804508 16336 log.cc:1196] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/32be767b586348c48a049fc5bef277cf
I20260620 14:51:54.804771 16336 ts_tablet_manager.cc:1939] T 32be767b586348c48a049fc5bef277cf P 67cd19de53d94e0fac47e9f0911e5ddd: Deleting consensus metadata
I20260620 14:51:54.805186 15823 catalog_manager.cc:4841] TS 67cd19de53d94e0fac47e9f0911e5ddd (127.5.118.131:36755): tablet 32be767b586348c48a049fc5bef277cf (table test-table [id=7024dbc3ae9c4aee81f98ba09ed52cb4]) successfully deleted
W20260620 14:51:54.806782 15824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:39067 (error 108)
I20260620 14:51:54.807346 16309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.807209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39067 (local address 127.0.0.1:45066)
0620 14:51:54.807266 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.807312 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:51:54.807427 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 16033
W20260620 14:51:54.807459 15824 catalog_manager.cc:4584] TS eb7563e30f0e4094a97634dea86e4929 (127.5.118.129:39067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32be767b586348c48a049fc5bef277cf: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39067: connect: Connection refused (error 111)
I20260620 14:51:54.811681 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 16167
I20260620 14:51:54.815863 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 15805
2026-06-20T14:51:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2194 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:54 dist-test-slave-2m5f krb5kdc[16342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:54 dist-test-slave-2m5f krb5kdc[16342](info): set up 2 sockets
Jun 20 14:51:54 dist-test-slave-2m5f krb5kdc[16342](info): commencing operation
krb5kdc: starting...
W20260620 14:51:55.889312 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:51:55 dist-test-slave-2m5f krb5kdc[16342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.118.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.118.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/rangeradmin_127.5.118.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/rangeradmin_127.5.118.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.118.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.118.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/rangerlookup_127.5.118.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/rangerlookup_127.5.118.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.118.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.118.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/HTTP_127.5.118.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/HTTP_127.5.118.148@KRBTEST.COM.keytab.
I20260620 14:51:56.066586 5594 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjla6OK/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/postgres -l logfile start
2026-06-20 14:51:58.603 UTC [16372] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:51:58.603 UTC [16372] LOG: listening on IPv4 address "127.5.118.148", port 36481
2026-06-20 14:51:58.608 UTC [16372] LOG: listening on Unix socket "/tmp/.s.PGSQL.36481"
2026-06-20 14:51:58.643 UTC [16375] LOG: database system was shut down at 2026-06-20 14:51:57 UTC
2026-06-20 14:51:58.647 UTC [16372] LOG: database system is ready to accept connections
I20260620 14:51:59.597126 5594 mini_postgres.cc:96] Postgres bound to 36481
2026-06-20 14:51:59.603 UTC [16383] FATAL: database "slave" does not exist
127.5.118.148:36481 - accepting connections
I20260620 14:51:59.604043 5594 mini_ranger.cc:161] Starting Ranger...
I20260620 14:51:59.619457 5594 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:52:00.132972 5594 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:52:00.133073 5594 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-admin
2026-06-20 14:52:00,415 [I] DB FLAVOR :POSTGRES
2026-06-20 14:52:00,415 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:52:00,416 [I] Checking connection..
2026-06-20 14:52:00,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:52:00,687 [I] Checking connection passed.
2026-06-20 14:52:00,687 [I] --------- Verifying version history table ---------
2026-06-20 14:52:00,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:52:00,957 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:52:00,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:52:01,192 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:52:01,192 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:52:01,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:52:01.455 UTC [16447] WARNING: there is no transaction in progress
2026-06-20 14:52:01,462 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:52:01,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:52:01,709 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:52:01,709 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:52:01,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:52:01,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:52:02,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2m5f.c.gcp-upstream.internal','N') ;"
2026-06-20 14:52:02,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:52:02,703 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:52:02,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:52:02,954 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:52:02,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:52:03,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:52:03,453 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:52:03,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:52:04.590 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.610 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.632 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.652 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.671 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.695 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.771 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.780 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.798 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.811 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.825 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.842 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.854 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.863 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.872 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:04.881 UTC [16555] WARNING: there is no transaction in progress
2026-06-20 14:52:05,702 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:52:05,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:52:05,964 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:52:05,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:52:06,230 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:52:06,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:52:06,498 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:52:06,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:52:06,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:52:06,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2m5f.c.gcp-upstream.internal';"
2026-06-20 14:52:07,213 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:52:07,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:52:07,461 [I] DB_PATCHES have already been applied
I20260620 14:52:07.467530 5594 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjla6OK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:52:07.467598 5594 mini_ranger.cc:193] Using host: 127.5.118.148
I20260620 14:52:07.470767 5594 mini_ranger.cc:227] Ranger admin URL: http://127.5.118.148:57165
Jun 20, 2026 2:52:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:52:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:52:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:52:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:52:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:52:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:52:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:52:08.593966 5594 mini_ranger.cc:160] Time spent starting Ranger: real 8.990s user 0.000s sys 0.010s
Jun 20, 2026 2:52:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:52:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 2:52:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:52:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:52:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:52:31.209450 5594 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.118.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.118.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/rangerkms_127.5.118.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/rangerkms_127.5.118.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.118.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.118.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.118.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/HTTP_127.5.118.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.118.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/HTTP_127.5.118.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:52:31 dist-test-slave-2m5f krb5kdc[16342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967151, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:52:31.321548 5594 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:52:31.332728 5594 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:52:32.383975 5594 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:52:32.384068 5594 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-kms
I20260620 14:52:32.384090 5594 mini_ranger_kms.cc:227] Using postgres at 127.5.118.148:36481
2026-06-20 14:52:32,567 [I] DB FLAVOR :POSTGRES
2026-06-20 14:52:32,567 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:52:32,567 [I] Checking connection
2026-06-20 14:52:32,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:52:32,807 [I] connection success
2026-06-20 14:52:32,807 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:52:32,807 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:52:32,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:52:33,078 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:52:33,078 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:52:33,078 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:52:33,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:36481/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:52:33,366 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:52:33.809119 5594 mini_ranger_kms.cc:310] Created kms service
I20260620 14:52:33.992504 5594 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:52:34.046010 5594 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:52:34 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967151, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:52:34.532068 5594 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:52:34.532318 5594 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:52:34.532357 5594 mini_ranger_kms.cc:240] Using host: 127.5.118.148
I20260620 14:52:34.535575 5594 mini_ranger_kms.cc:276] Ranger KMS PID: 16736
I20260620 14:52:34.535710 5594 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.118.148:50845
Jun 20, 2026 2:52:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:52:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:52:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:52:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:52:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:52:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:52:36.850317 5594 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.529s user 0.013s sys 0.152s
I20260620 14:52:36.850441 5594 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:52:36.850555 5594 mini_ranger_kms.cc:401] 127.5.118.148:50845/kms/v1/keys
Jun 20 14:52:37 dist-test-slave-2m5f krb5kdc[16342](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967157, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.118.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:52:37 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967157, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.118.148@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
Jun 20 14:52:37 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967157, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.118.148@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
Jun 20 14:52:40 dist-test-slave-2m5f krb5kdc[16342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:52:40.220220 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43959
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:43959
--ranger_config_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:50845/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:40.302685 16786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:40.302893 16786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:40.302915 16786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:40.304054 16786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:40.304101 16786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:40.304116 16786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:40.304129 16786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:40.304141 16786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:40.305747 16786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.118.148:50845/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:43959
--ranger_config_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:52:40.305944 16786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:40.306123 16786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:40.311497 16791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:40.311579 16792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:40.311540 16786 server_base.cc:1034] running on GCE node
W20260620 14:52:40.311784 16794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:40.312052 16786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:40.312403 16786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:40.313575 16786 hybrid_clock.cc:648] HybridClock initialized: now 1781967160313565 us; error 39 us; skew 500 ppm
Jun 20 14:52:40 dist-test-slave-2m5f krb5kdc[16342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967160, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:40.316113 16786 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:52:40.316630 16786 webserver.cc:460] Webserver started at http://127.5.118.190:36091/ using document root <none> and password file <none>
I20260620 14:52:40.316869 16786 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:40.316928 16786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:40.317085 16786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:52:40 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967160, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:52:40.376407 16786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "3558a903adba49c798e1a87e9125ae2a"
format_stamp: "Formatted at 2026-06-20 14:52:40 on dist-test-slave-2m5f"
server_key: "83cea806036c0310978fddc717679908"
server_key_iv: "1465e7f39197dcaf10b3e0a324767582"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:40.376854 16786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "3558a903adba49c798e1a87e9125ae2a"
format_stamp: "Formatted at 2026-06-20 14:52:40 on dist-test-slave-2m5f"
server_key: "83cea806036c0310978fddc717679908"
server_key_iv: "1465e7f39197dcaf10b3e0a324767582"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:40.378754 16786 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:52:40.394503 16801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:40.394833 16786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:52:40.394950 16786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "3558a903adba49c798e1a87e9125ae2a"
format_stamp: "Formatted at 2026-06-20 14:52:40 on dist-test-slave-2m5f"
server_key: "83cea806036c0310978fddc717679908"
server_key_iv: "1465e7f39197dcaf10b3e0a324767582"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:40.395056 16786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:40.410218 16786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:40.412506 16786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:40.412750 16786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:40.417552 16786 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:43959
I20260620 14:52:40.417599 16853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:43959 every 8 connection(s)
I20260620 14:52:40.417979 16786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:40.418658 16854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:40.421084 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a: Bootstrap starting.
I20260620 14:52:40.421609 16854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:40.421846 16854 log.cc:824] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:40.422503 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a: No bootstrap required, opened a new log
I20260620 14:52:40.424554 16854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3558a903adba49c798e1a87e9125ae2a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43959 } }
I20260620 14:52:40.424702 16854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:40.424727 16854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3558a903adba49c798e1a87e9125ae2a, State: Initialized, Role: FOLLOWER
I20260620 14:52:40.424830 16854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3558a903adba49c798e1a87e9125ae2a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43959 } }
I20260620 14:52:40.424909 16854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:40.424944 16854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:40.425015 16854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:40.425554 16854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3558a903adba49c798e1a87e9125ae2a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43959 } }
I20260620 14:52:40.425671 16854 leader_election.cc:302] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3558a903adba49c798e1a87e9125ae2a; no voters:
I20260620 14:52:40.425894 16854 leader_election.cc:288] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:40.425935 16857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:40.426174 16854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:40.426179 16857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [term 1 LEADER]: Becoming Leader. State: Replica: 3558a903adba49c798e1a87e9125ae2a, State: Running, Role: LEADER
I20260620 14:52:40.426383 16857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3558a903adba49c798e1a87e9125ae2a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43959 } }
I20260620 14:52:40.426501 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 16786
I20260620 14:52:40.426615 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:52:40.426803 16859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3558a903adba49c798e1a87e9125ae2a. Latest consensus state: current_term: 1 leader_uuid: "3558a903adba49c798e1a87e9125ae2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3558a903adba49c798e1a87e9125ae2a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43959 } } }
I20260620 14:52:40.426863 5594 external_mini_cluster.cc:1428] Setting key a9e4822c2946293abda5f7ed3d4db322
I20260620 14:52:40.426887 16859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:40.427145 16858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3558a903adba49c798e1a87e9125ae2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3558a903adba49c798e1a87e9125ae2a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43959 } } }
I20260620 14:52:40.427278 16858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:40.427449 16866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:40.427736 16854 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:52:40.428016 16866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:40.434345 16866 catalog_manager.cc:1347] Generated new cluster ID: 50dd26ab66b7416bbc974ff1e9633e1c
I20260620 14:52:40.434469 16866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:52:40 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:52:40.441936 16872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:40.427374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.5.118.190:43959)
0620 14:52:40.427733 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:40.427744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:40.428416 (+ 672us) server_negotiation.cc:407] Connection header received
0620 14:52:40.430032 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:40.430085 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:40.430273 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:40.430485 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:40.431688 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:40.433603 (+ 1915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:40.434710 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:40.435348 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:40.435466 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:40.438190 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:40.438195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:40.438195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:40.438207 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:40.440076 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:40.440460 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:40.440463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:40.440464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:40.440568 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:40.440751 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:40.440751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:40.440752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:40.441052 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:40.441205 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:40.441599 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 14:52:40.441712 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":306,"thread_start_us":61,"threads_started":1}
I20260620 14:52:40.453135 16866 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:40.453724 16866 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:40.461544 16866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a: Generated new TSK 0
I20260620 14:52:40.461767 16866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:52:41.171875 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:50845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43959
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:41.283412 16899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:41.283661 16899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:41.283703 16899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:41.285018 16899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:41.285092 16899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:41.285135 16899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:52:41.286644 16899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.118.148:50845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:52:41.286918 16899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:41.287196 16899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:41.293838 16905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:41.293853 16904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:41.293951 16907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:41.294049 16899 server_base.cc:1034] running on GCE node
I20260620 14:52:41.294309 16899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:41.294627 16899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:41.295828 16899 hybrid_clock.cc:648] HybridClock initialized: now 1781967161295795 us; error 46 us; skew 500 ppm
Jun 20 14:52:41 dist-test-slave-2m5f krb5kdc[16342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:41.299019 16899 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:52:41.299621 16899 webserver.cc:460] Webserver started at http://127.5.118.129:43145/ using document root <none> and password file <none>
I20260620 14:52:41.299849 16899 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:41.299885 16899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:41.299979 16899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:52:41 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:52:41.315500 16899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "f43e08fb144e4c469acd597b9056d6d2"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-2m5f"
server_key: "c7b3911b19085eb1786918cf69c06a38"
server_key_iv: "05dcb6aa81d334ef155f79110be3e2dd"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:41.316037 16899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "f43e08fb144e4c469acd597b9056d6d2"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-2m5f"
server_key: "c7b3911b19085eb1786918cf69c06a38"
server_key_iv: "05dcb6aa81d334ef155f79110be3e2dd"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:41.321561 16899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260620 14:52:41.341418 16914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:41.341702 16899 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 14:52:41.341792 16899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "f43e08fb144e4c469acd597b9056d6d2"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-2m5f"
server_key: "c7b3911b19085eb1786918cf69c06a38"
server_key_iv: "05dcb6aa81d334ef155f79110be3e2dd"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:41.341905 16899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:41.399802 16899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:41.402348 16899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:41.402540 16899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:41.402844 16899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:41.403246 16899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:41.403344 16899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:41.403420 16899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:41.403462 16899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:41.411623 16899 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:40213
I20260620 14:52:41.412030 16899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:41.412710 17029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:40213 every 8 connection(s)
I20260620 14:52:41.418998 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 16899
I20260620 14:52:41.419124 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:41.419341 5594 external_mini_cluster.cc:1428] Setting key ed99bb313322749b524332e543ea4012
Jun 20 14:52:41 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:52:41.427353 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.413135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:60929 (local address 127.5.118.190:43959)
0620 14:52:41.413289 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:41.413291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:41.413521 (+ 230us) server_negotiation.cc:407] Connection header received
0620 14:52:41.414121 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:41.414123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:41.414139 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:41.414204 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:41.414935 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.415501 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.417076 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.417322 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.417411 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.422783 (+ 5372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:41.422785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:41.422786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:41.422796 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:41.425081 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:41.425785 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:41.425787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:41.425788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:41.425866 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:41.426219 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:41.426220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:41.426221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:41.426409 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:41.426466 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:41.427064 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 14:52:41.427183 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260620 14:52:41.428395 17032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.413138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43959 (local address 127.5.118.129:60929)
0620 14:52:41.413393 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:41.413427 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:41.413988 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:41.414299 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:41.414306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:41.414619 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:41.414831 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.414837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.415612 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.415613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.416971 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.416974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.419680 (+ 2706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.419681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.419863 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.420597 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:41.420609 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:41.422633 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:41.425215 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:41.425226 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:41.425229 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:41.425648 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:41.425949 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:41.425950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:41.425951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:41.426129 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:41.426523 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:41.426530 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:41.426856 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 14:52:41.427840 (+ 984us) client_negotiation.cc:240] Negotiation successful
0620 14:52:41.428054 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":75,"threads_started":1}
I20260620 14:52:41.429399 17030 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43959
I20260620 14:52:41.429507 17030 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:41.429873 17030 heartbeater.cc:511] Master 127.5.118.190:43959 requested a full tablet report, sending...
I20260620 14:52:41.430624 16818 ts_manager.cc:194] Registered new tserver with Master: f43e08fb144e4c469acd597b9056d6d2 (127.5.118.129:40213)
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
I20260620 14:52:41.431604 16818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:60929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:52:41.452983 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:50845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43959
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:52:41 dist-test-slave-2m5f krb5kdc[16342](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:52:41.550563 17038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:41.550951 17038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:41.551033 17038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:41.552950 17038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:41.553128 17038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:41.553215 17038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:52:41.555586 17038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.118.148:50845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:52:41.556162 17038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:41.556546 17038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:41.565122 17046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:41.565059 17044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:41.565557 17038 server_base.cc:1034] running on GCE node
W20260620 14:52:41.568432 17043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:41.568843 17038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:41.569166 17038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:41.570366 17038 hybrid_clock.cc:648] HybridClock initialized: now 1781967161570330 us; error 50 us; skew 500 ppm
Jun 20 14:52:41 dist-test-slave-2m5f krb5kdc[16342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:41.573911 17038 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:52:41.574518 17038 webserver.cc:460] Webserver started at http://127.5.118.130:43499/ using document root <none> and password file <none>
I20260620 14:52:41.574777 17038 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:41.574829 17038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:41.574955 17038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:52:41 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:52:41.587836 17038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "e648f450cf9d406d812cc4392c3c4c72"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-2m5f"
server_key: "cfda009a45c4cbac2c7a08bb450003a2"
server_key_iv: "a511b88c3443b83ea75f36264931d007"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:41.588328 17038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "e648f450cf9d406d812cc4392c3c4c72"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-2m5f"
server_key: "cfda009a45c4cbac2c7a08bb450003a2"
server_key_iv: "a511b88c3443b83ea75f36264931d007"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:41.590147 17038 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:52:41.603861 17053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:41.604104 17038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:52:41.604265 17038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "e648f450cf9d406d812cc4392c3c4c72"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-2m5f"
server_key: "cfda009a45c4cbac2c7a08bb450003a2"
server_key_iv: "a511b88c3443b83ea75f36264931d007"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:41.604384 17038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:41.633714 17038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:41.636376 17038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:41.636593 17038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:41.636924 17038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:41.637367 17038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:41.637463 17038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:41.637538 17038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:41.637598 17038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:41.645152 17038 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:35701
I20260620 14:52:41.645707 17038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:41.647504 17167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:35701 every 8 connection(s)
I20260620 14:52:41.653242 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 17038
I20260620 14:52:41.653357 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:41.653548 5594 external_mini_cluster.cc:1428] Setting key e5f02ab06feee186065022916f2a2988
Jun 20 14:52:41 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:52:41.663944 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.649030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:35881 (local address 127.5.118.190:43959)
0620 14:52:41.649073 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:41.649075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:41.649085 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:52:41.649772 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:41.649774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:41.649791 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:41.649823 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:41.650696 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.651458 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.654091 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.654415 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.654468 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.659971 (+ 5503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:41.659974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:41.659974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:41.659984 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:41.661988 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:41.662646 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:41.662648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:41.662649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:41.662717 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:41.663041 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:41.663042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:41.663042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:41.663220 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:41.663272 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:41.663719 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 14:52:41.663820 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:52:41.664907 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.648299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43959 (local address 127.5.118.130:35881)
0620 14:52:41.648929 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:41.648979 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:41.649638 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:41.650021 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:41.650026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:41.650293 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:41.650570 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.650576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.651593 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.651594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.652918 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.652921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.656325 (+ 3404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.656326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.656537 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.657979 (+ 1442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:41.657992 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:41.659822 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:41.662123 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:41.662134 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:41.662138 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:41.662536 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:41.662795 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:41.662796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:41.662796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:41.662963 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:41.663325 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:41.663332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:41.663541 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 14:52:41.664387 (+ 846us) client_negotiation.cc:240] Negotiation successful
0620 14:52:41.664597 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
I20260620 14:52:41.665676 17168 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43959
I20260620 14:52:41.665778 17168 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:41.665979 17168 heartbeater.cc:511] Master 127.5.118.190:43959 requested a full tablet report, sending...
I20260620 14:52:41.666424 16818 ts_manager.cc:194] Registered new tserver with Master: e648f450cf9d406d812cc4392c3c4c72 (127.5.118.130:35701)
I20260620 14:52:41.666908 16818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:35881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:52:41.687577 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:50845/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43959
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:41.816769 17173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:41.817087 17173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:41.817133 17173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:41.819063 17173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:41.819149 17173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:41.819192 17173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:52:41.821764 17173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.118.148:50845/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:52:41.822046 17173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:41.822391 17173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:41.834023 17178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:41.834096 17181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:41.834132 17179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:41.834340 17173 server_base.cc:1034] running on GCE node
I20260620 14:52:41.834597 17173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:41.834899 17173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:41.836128 17173 hybrid_clock.cc:648] HybridClock initialized: now 1781967161835902 us; error 241 us; skew 500 ppm
Jun 20 14:52:41 dist-test-slave-2m5f krb5kdc[16342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:41.839247 17173 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:52:41.839848 17173 webserver.cc:460] Webserver started at http://127.5.118.131:36027/ using document root <none> and password file <none>
I20260620 14:52:41.840106 17173 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:41.840221 17173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:41.840411 17173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:52:41 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:52:41.856398 17173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "83cdb12d46754dda8554e5806df94062"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-2m5f"
server_key: "de38907e4a3b7d19885d2d334c20c4bb"
server_key_iv: "0c7273d76e59dba25218ab5f1aecd348"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:41.856981 17173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "83cdb12d46754dda8554e5806df94062"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-2m5f"
server_key: "de38907e4a3b7d19885d2d334c20c4bb"
server_key_iv: "0c7273d76e59dba25218ab5f1aecd348"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:41.859504 17173 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 14:52:41.901418 17188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:41.901649 17173 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260620 14:52:41.901767 17173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "83cdb12d46754dda8554e5806df94062"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-2m5f"
server_key: "de38907e4a3b7d19885d2d334c20c4bb"
server_key_iv: "0c7273d76e59dba25218ab5f1aecd348"
server_key_version: "kuduclusterkey@0"
I20260620 14:52:41.901894 17173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:41.923134 17173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:41.925760 17173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:41.925930 17173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:41.926185 17173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:41.926587 17173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:41.926656 17173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:41.926700 17173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:41.926728 17173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:41.935050 17173 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:37925
I20260620 14:52:41.935767 17173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:41.940809 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 17173
I20260620 14:52:41.940914 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:41.941092 5594 external_mini_cluster.cc:1428] Setting key f412ba5460115733a2770719660aee91
Jun 20 14:52:41 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:52:41.946820 17301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:37925 every 8 connection(s)
I20260620 14:52:41.948604 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.936699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:38163 (local address 127.5.118.190:43959)
0620 14:52:41.936780 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:41.936782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:41.937469 (+ 687us) server_negotiation.cc:407] Connection header received
0620 14:52:41.938366 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:41.938368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:41.938382 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:41.938454 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:41.939213 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.940015 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.941797 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.942102 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.942151 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.945034 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:41.945037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:41.945037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:41.945046 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:41.946816 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:41.947331 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:41.947332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:41.947333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:41.947404 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:41.947791 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:41.947792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:41.947792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:41.947937 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:41.948009 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:41.948420 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 14:52:41.948510 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:52:41.948796 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.936639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43959 (local address 127.5.118.131:38163)
0620 14:52:41.937311 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:41.937368 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:41.938235 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:41.938537 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:41.938542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:41.938823 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:41.939112 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.939119 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.940114 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.940116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.941697 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.941700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.942250 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.942251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.942416 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.943070 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:41.943082 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:41.944915 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:41.946911 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:41.946917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:41.946920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:41.947244 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:41.947483 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:41.947484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:41.947484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:41.947718 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:41.948018 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:41.948024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:41.948251 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 14:52:41.948375 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 14:52:41.948547 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":554,"threads_started":1}
I20260620 14:52:41.949370 17302 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43959
I20260620 14:52:41.949471 17302 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:41.949677 17302 heartbeater.cc:511] Master 127.5.118.190:43959 requested a full tablet report, sending...
I20260620 14:52:41.950098 16818 ts_manager.cc:194] Registered new tserver with Master: 83cdb12d46754dda8554e5806df94062 (127.5.118.131:37925)
I20260620 14:52:41.950625 16818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:38163
I20260620 14:52:41.952761 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:41.964757 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.956597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.5.118.190:43959)
0620 14:52:41.956650 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:41.956652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:41.956908 (+ 256us) server_negotiation.cc:407] Connection header received
0620 14:52:41.957007 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:41.957008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:41.957018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:41.957048 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:41.957865 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.958650 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.959792 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.960092 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.960133 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.961773 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:41.961775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:41.961775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:41.961783 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:41.963477 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:41.963891 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:41.963892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:41.963893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:41.963946 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:41.964127 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:41.964128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:41.964128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:41.964292 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:41.964371 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:41.964574 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 14:52:41.964655 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:52:41.967058 16818 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:41.969573 16818 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:41.978675 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.973210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.5.118.131:37925)
0620 14:52:41.973671 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:41.973676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:41.973961 (+ 285us) server_negotiation.cc:407] Connection header received
0620 14:52:41.974083 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:41.974086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:41.974102 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:41.974159 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:41.974387 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.975485 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.977247 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.978058 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.978114 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.978202 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:41.978515 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 14:52:41.978561 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":94,"threads_started":1}
I20260620 14:52:41.978694 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.972745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:35701 (local address 127.0.0.1:37200)
0620 14:52:41.972925 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:41.972974 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:41.973058 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:41.973939 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:41.973942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:41.973945 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:41.974021 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.974023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.975725 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.975726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.977080 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.977083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.978447 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.978448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.978515 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.978519 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:52:41.978548 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 14:52:41.978611 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260620 14:52:41.978965 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.973090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:37925 (local address 127.0.0.1:37396)
0620 14:52:41.973900 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:41.973909 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:41.973990 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:41.974264 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:41.974264 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:41.974266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:41.974311 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.974313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.975610 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.975611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.977091 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.977095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.978268 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.978269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.978372 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.978394 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:52:41.978827 (+ 433us) client_negotiation.cc:240] Negotiation successful
0620 14:52:41.978879 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":55,"thread_start_us":47,"threads_started":1}
I20260620 14:52:41.979487 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.973577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.5.118.130:35701)
0620 14:52:41.973755 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:41.973759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:41.973768 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:52:41.973809 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:41.973811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:41.973828 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:41.974090 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:41.974173 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.975637 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.977227 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.978346 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.979241 (+ 895us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.979337 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:41.979365 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 14:52:41.979408 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260620 14:52:41.980044 17083 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6063904126468c86da49bd18b434fe (DEFAULT_TABLE table=test-table [id=2171eaa9f9794187b3bcaa800eb2e2fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:41.980060 17236 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6063904126468c86da49bd18b434fe (DEFAULT_TABLE table=test-table [id=2171eaa9f9794187b3bcaa800eb2e2fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:41.980437 17236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6063904126468c86da49bd18b434fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:41.980438 17083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6063904126468c86da49bd18b434fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:41.985021 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.972952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:40213 (local address 127.0.0.1:46184)
0620 14:52:41.973755 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:41.973767 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:41.973820 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:41.980807 (+ 6987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:41.980808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:41.980811 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:41.980857 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.980858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.982150 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.982151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.983251 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:41.983255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.984794 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.984795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:41.984863 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.984868 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:52:41.984898 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 14:52:41.984941 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":63,"threads_started":1}
I20260620 14:52:41.985440 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.973449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46184 (local address 127.5.118.129:40213)
0620 14:52:41.980628 (+ 7179us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:41.980633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:41.980644 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:52:41.980680 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:41.980682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:41.980700 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:41.980933 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:41.980998 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.982038 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.983366 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:41.984693 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:41.985211 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:41.985287 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:41.985314 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:52:41.985361 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7139,"thread_start_us":64,"threads_started":1}
I20260620 14:52:41.985817 17322 tablet_bootstrap.cc:492] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062: Bootstrap starting.
I20260620 14:52:41.986146 16964 tablet_service.cc:1467] Processing CreateTablet for tablet 9d6063904126468c86da49bd18b434fe (DEFAULT_TABLE table=test-table [id=2171eaa9f9794187b3bcaa800eb2e2fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:41.986544 16964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d6063904126468c86da49bd18b434fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:41.986819 17322 tablet_bootstrap.cc:654] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:41.987119 17322 log.cc:824] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:41.987484 17321 tablet_bootstrap.cc:492] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72: Bootstrap starting.
I20260620 14:52:41.988976 17321 tablet_bootstrap.cc:654] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:41.989292 17321 log.cc:824] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:41.990137 17322 tablet_bootstrap.cc:492] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062: No bootstrap required, opened a new log
I20260620 14:52:41.990227 17322 ts_tablet_manager.cc:1397] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:52:41.990742 17325 tablet_bootstrap.cc:492] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2: Bootstrap starting.
I20260620 14:52:41.991551 17325 tablet_bootstrap.cc:654] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:41.991979 17325 log.cc:824] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:41.992128 17322 raft_consensus.cc:348] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e648f450cf9d406d812cc4392c3c4c72" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35701 } } peers { permanent_uuid: "f43e08fb144e4c469acd597b9056d6d2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40213 } } peers { permanent_uuid: "83cdb12d46754dda8554e5806df94062" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37925 } }
I20260620 14:52:41.992345 17322 raft_consensus.cc:374] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:41.992385 17322 raft_consensus.cc:729] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83cdb12d46754dda8554e5806df94062, State: Initialized, Role: FOLLOWER
I20260620 14:52:41.992524 17322 consensus_queue.cc:260] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e648f450cf9d406d812cc4392c3c4c72" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35701 } } peers { permanent_uuid: "f43e08fb144e4c469acd597b9056d6d2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40213 } } peers { permanent_uuid: "83cdb12d46754dda8554e5806df94062" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37925 } }
I20260620 14:52:41.992779 17322 ts_tablet_manager.cc:1428] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:41.992897 17302 heartbeater.cc:503] Master 127.5.118.190:43959 was elected leader, sending a full tablet report...
I20260620 14:52:41.994593 17321 tablet_bootstrap.cc:492] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72: No bootstrap required, opened a new log
I20260620 14:52:41.994678 17321 ts_tablet_manager.cc:1397] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 14:52:41.997251 17325 tablet_bootstrap.cc:492] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2: No bootstrap required, opened a new log
I20260620 14:52:41.997335 17325 ts_tablet_manager.cc:1397] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 14:52:41.997475 17321 raft_consensus.cc:348] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e648f450cf9d406d812cc4392c3c4c72" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35701 } } peers { permanent_uuid: "f43e08fb144e4c469acd597b9056d6d2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40213 } } peers { permanent_uuid: "83cdb12d46754dda8554e5806df94062" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37925 } }
I20260620 14:52:41.997654 17321 raft_consensus.cc:374] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:41.997687 17321 raft_consensus.cc:729] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e648f450cf9d406d812cc4392c3c4c72, State: Initialized, Role: FOLLOWER
I20260620 14:52:41.997825 17321 consensus_queue.cc:260] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e648f450cf9d406d812cc4392c3c4c72" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35701 } } peers { permanent_uuid: "f43e08fb144e4c469acd597b9056d6d2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40213 } } peers { permanent_uuid: "83cdb12d46754dda8554e5806df94062" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37925 } }
I20260620 14:52:41.998090 17321 ts_tablet_manager.cc:1428] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:52:41.998204 17168 heartbeater.cc:503] Master 127.5.118.190:43959 was elected leader, sending a full tablet report...
I20260620 14:52:42.000537 17325 raft_consensus.cc:348] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e648f450cf9d406d812cc4392c3c4c72" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35701 } } peers { permanent_uuid: "f43e08fb144e4c469acd597b9056d6d2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40213 } } peers { permanent_uuid: "83cdb12d46754dda8554e5806df94062" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37925 } }
I20260620 14:52:42.000739 17325 raft_consensus.cc:374] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:42.000782 17325 raft_consensus.cc:729] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f43e08fb144e4c469acd597b9056d6d2, State: Initialized, Role: FOLLOWER
I20260620 14:52:42.000919 17325 consensus_queue.cc:260] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e648f450cf9d406d812cc4392c3c4c72" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35701 } } peers { permanent_uuid: "f43e08fb144e4c469acd597b9056d6d2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40213 } } peers { permanent_uuid: "83cdb12d46754dda8554e5806df94062" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37925 } }
I20260620 14:52:42.001215 17325 ts_tablet_manager.cc:1428] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 14:52:42.001255 17030 heartbeater.cc:503] Master 127.5.118.190:43959 was elected leader, sending a full tablet report...
Jun 20 14:52:42 dist-test-slave-2m5f krb5kdc[16342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:52:42.049635 17329 raft_consensus.cc:482] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:42.049788 17329 raft_consensus.cc:504] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e648f450cf9d406d812cc4392c3c4c72" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35701 } } peers { permanent_uuid: "f43e08fb144e4c469acd597b9056d6d2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40213 } } peers { permanent_uuid: "83cdb12d46754dda8554e5806df94062" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37925 } }
I20260620 14:52:42.050251 17329 leader_election.cc:288] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e648f450cf9d406d812cc4392c3c4c72 (127.5.118.130:35701), 83cdb12d46754dda8554e5806df94062 (127.5.118.131:37925)
I20260620 14:52:42.054733 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:42.050917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:46733 (local address 127.5.118.130:35701)
0620 14:52:42.051026 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:42.051028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:42.051208 (+ 180us) server_negotiation.cc:407] Connection header received
0620 14:52:42.051314 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:42.051315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:42.051324 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:42.051353 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:42.051587 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:42.052324 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:42.053757 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:42.054315 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:42.054373 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:42.054384 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:42.054609 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:52:42.054654 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:52:42.055022 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:42.050505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:35701 (local address 127.5.118.129:46733)
0620 14:52:42.051137 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:42.051148 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:42.051236 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:42.051421 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:42.051422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:42.051423 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:42.051507 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:42.051509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:42.052440 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:42.052441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:42.053640 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:42.053644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:42.054449 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:42.054449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:42.054522 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:42.054526 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:52:42.054875 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 14:52:42.054920 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":91,"threads_started":1}
I20260620 14:52:42.055810 17122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6063904126468c86da49bd18b434fe" candidate_uuid: "f43e08fb144e4c469acd597b9056d6d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e648f450cf9d406d812cc4392c3c4c72" is_pre_election: true
I20260620 14:52:42.055984 17122 raft_consensus.cc:2454] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f43e08fb144e4c469acd597b9056d6d2 in term 0.
I20260620 14:52:42.056250 16920 leader_election.cc:302] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e648f450cf9d406d812cc4392c3c4c72, f43e08fb144e4c469acd597b9056d6d2; no voters:
I20260620 14:52:42.056387 17329 raft_consensus.cc:2781] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:42.056427 17329 raft_consensus.cc:482] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:42.056450 17329 raft_consensus.cc:3037] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:42.057406 17329 raft_consensus.cc:504] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e648f450cf9d406d812cc4392c3c4c72" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35701 } } peers { permanent_uuid: "f43e08fb144e4c469acd597b9056d6d2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40213 } } peers { permanent_uuid: "83cdb12d46754dda8554e5806df94062" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37925 } }
I20260620 14:52:42.057547 17329 leader_election.cc:288] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [CANDIDATE]: Term 1 election: Requested vote from peers e648f450cf9d406d812cc4392c3c4c72 (127.5.118.130:35701), 83cdb12d46754dda8554e5806df94062 (127.5.118.131:37925)
I20260620 14:52:42.057768 17122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6063904126468c86da49bd18b434fe" candidate_uuid: "f43e08fb144e4c469acd597b9056d6d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e648f450cf9d406d812cc4392c3c4c72"
I20260620 14:52:42.057847 17122 raft_consensus.cc:3037] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:42.058770 17122 raft_consensus.cc:2454] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f43e08fb144e4c469acd597b9056d6d2 in term 1.
I20260620 14:52:42.058988 16920 leader_election.cc:302] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e648f450cf9d406d812cc4392c3c4c72, f43e08fb144e4c469acd597b9056d6d2; no voters:
I20260620 14:52:42.059289 17329 raft_consensus.cc:2781] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:42.059490 17329 raft_consensus.cc:686] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [term 1 LEADER]: Becoming Leader. State: Replica: f43e08fb144e4c469acd597b9056d6d2, State: Running, Role: LEADER
I20260620 14:52:42.059664 17329 consensus_queue.cc:237] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e648f450cf9d406d812cc4392c3c4c72" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35701 } } peers { permanent_uuid: "f43e08fb144e4c469acd597b9056d6d2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40213 } } peers { permanent_uuid: "83cdb12d46754dda8554e5806df94062" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37925 } }
I20260620 14:52:42.060707 16817 catalog_manager.cc:5495] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f43e08fb144e4c469acd597b9056d6d2 (127.5.118.129). New cstate: current_term: 1 leader_uuid: "f43e08fb144e4c469acd597b9056d6d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e648f450cf9d406d812cc4392c3c4c72" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f43e08fb144e4c469acd597b9056d6d2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83cdb12d46754dda8554e5806df94062" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37925 } health_report { overall_health: UNKNOWN } } }
I20260620 14:52:42.068554 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:42.050965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:50231 (local address 127.5.118.131:37925)
0620 14:52:42.051062 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:42.051064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:42.052239 (+ 1175us) server_negotiation.cc:407] Connection header received
0620 14:52:42.052348 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:42.052349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:42.052360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:42.052391 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:42.065277 (+ 12886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:42.066028 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:42.067525 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:42.068078 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:42.068126 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:42.068135 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:42.068408 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 14:52:42.068476 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:52:42.068846 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:42.050698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:37925 (local address 127.5.118.129:50231)
0620 14:52:42.052128 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:42.052139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:42.052265 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:42.063793 (+ 11528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:42.063794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:42.063796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:42.063929 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:42.063932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:42.066162 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:42.066163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:42.067423 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:42.067426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:42.068251 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:42.068252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:42.068327 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:42.068331 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:52:42.068698 (+ 367us) client_negotiation.cc:240] Negotiation successful
0620 14:52:42.068745 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1402,"thread_start_us":63,"threads_started":1}
I20260620 14:52:42.069242 17256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6063904126468c86da49bd18b434fe" candidate_uuid: "f43e08fb144e4c469acd597b9056d6d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83cdb12d46754dda8554e5806df94062" is_pre_election: true
I20260620 14:52:42.069410 17256 raft_consensus.cc:2454] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f43e08fb144e4c469acd597b9056d6d2 in term 0.
I20260620 14:52:42.069556 17255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d6063904126468c86da49bd18b434fe" candidate_uuid: "f43e08fb144e4c469acd597b9056d6d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83cdb12d46754dda8554e5806df94062"
I20260620 14:52:42.069633 17255 raft_consensus.cc:3037] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:42.070572 17255 raft_consensus.cc:2454] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f43e08fb144e4c469acd597b9056d6d2 in term 1.
I20260620 14:52:42.074513 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:42.066643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46186 (local address 127.5.118.129:40213)
0620 14:52:42.066692 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:42.066694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:42.066704 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:52:42.066804 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:42.066806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:42.066827 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:42.066899 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:42.067097 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:42.067624 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:42.073264 (+ 5640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:42.073623 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:42.073704 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:42.073931 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:42.074069 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:42.074148 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:42.074396 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:52:42.074419 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:52:42.082139 17255 raft_consensus.cc:1261] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062 [term 1 FOLLOWER]: Refusing update from remote peer f43e08fb144e4c469acd597b9056d6d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:42.082139 17122 raft_consensus.cc:1261] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72 [term 1 FOLLOWER]: Refusing update from remote peer f43e08fb144e4c469acd597b9056d6d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:42.082568 17332 consensus_queue.cc:1035] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83cdb12d46754dda8554e5806df94062" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:42.082873 17329 consensus_queue.cc:1035] T 9d6063904126468c86da49bd18b434fe P f43e08fb144e4c469acd597b9056d6d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e648f450cf9d406d812cc4392c3c4c72" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:42.089625 17335 mvcc.cc:205] Tried to move back new op lower bound from 7298937495885156352 to 7298937495797190656. Current Snapshot: MvccSnapshot[applied={T|T < 7298937495885156352}]
I20260620 14:52:42.091617 17336 mvcc.cc:205] Tried to move back new op lower bound from 7298937495885156352 to 7298937495797190656. Current Snapshot: MvccSnapshot[applied={T|T < 7298937495885156352}]
I20260620 14:52:42.102735 16817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:42.102866 16817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:42.103628 16817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3558a903adba49c798e1a87e9125ae2a: Sending DeleteTablet for 3 replicas of tablet 9d6063904126468c86da49bd18b434fe
I20260620 14:52:42.104067 17083 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d6063904126468c86da49bd18b434fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:42 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37200
I20260620 14:52:42.104223 16964 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d6063904126468c86da49bd18b434fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:42 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:46184
I20260620 14:52:42.104238 17236 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d6063904126468c86da49bd18b434fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:42 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37396
I20260620 14:52:42.104799 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 16899
I20260620 14:52:42.112358 17347 tablet_replica.cc:330] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062: stopping tablet replica
I20260620 14:52:42.112588 17345 tablet_replica.cc:330] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72: stopping tablet replica
I20260620 14:52:42.113950 17345 raft_consensus.cc:2229] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:42.114180 17347 raft_consensus.cc:2229] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:42.114369 17347 raft_consensus.cc:2258] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:42.114519 17345 raft_consensus.cc:2258] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:42.115166 17347 ts_tablet_manager.cc:1905] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:42.119078 17347 ts_tablet_manager.cc:1918] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:42.119254 17347 log.cc:1196] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/9d6063904126468c86da49bd18b434fe
I20260620 14:52:42.119258 17345 ts_tablet_manager.cc:1905] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:42.119927 17347 ts_tablet_manager.cc:1939] T 9d6063904126468c86da49bd18b434fe P 83cdb12d46754dda8554e5806df94062: Deleting consensus metadata
I20260620 14:52:42.121084 17345 ts_tablet_manager.cc:1918] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:42.121176 17345 log.cc:1196] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/9d6063904126468c86da49bd18b434fe
I20260620 14:52:42.121467 17345 ts_tablet_manager.cc:1939] T 9d6063904126468c86da49bd18b434fe P e648f450cf9d406d812cc4392c3c4c72: Deleting consensus metadata
I20260620 14:52:42.121629 16805 catalog_manager.cc:4841] TS 83cdb12d46754dda8554e5806df94062 (127.5.118.131:37925): tablet 9d6063904126468c86da49bd18b434fe (table test-table [id=2171eaa9f9794187b3bcaa800eb2e2fe]) successfully deleted
I20260620 14:52:42.121935 16805 catalog_manager.cc:4841] TS e648f450cf9d406d812cc4392c3c4c72 (127.5.118.130:35701): tablet 9d6063904126468c86da49bd18b434fe (table test-table [id=2171eaa9f9794187b3bcaa800eb2e2fe]) successfully deleted
W20260620 14:52:42.122215 16805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:40213 (error 108)
I20260620 14:52:42.122823 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 17038
I20260620 14:52:42.129427 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 17173
I20260620 14:52:42.139391 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:42.139206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:40213 (local address 127.0.0.1:46188)
0620 14:52:42.139302 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:42.139341 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:40213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260620 14:52:42.139665 16805 catalog_manager.cc:4584] TS f43e08fb144e4c469acd597b9056d6d2 (127.5.118.129:40213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d6063904126468c86da49bd18b434fe: Network error: Client connection negotiation failed: client connection to 127.5.118.129:40213: connect: Connection refused (error 111)
I20260620 14:52:42.139919 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 16786
I20260620 14:52:42.150507 5594 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:52:42.495409 5594 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:52:42.495518 5594 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:52:42.840660 5594 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:52:42.840 UTC [16372] LOG: received smart shutdown request
2026-06-20 14:52:42.844 UTC [16372] LOG: background worker "logical replication launcher" (PID 16381) exited with exit code 1
2026-06-20 14:52:42.845 UTC [16376] LOG: shutting down
2026-06-20 14:52:43.029 UTC [16372] LOG: database system is shut down
2026-06-20T14:52:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48327 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:43 dist-test-slave-2m5f krb5kdc[17357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:43 dist-test-slave-2m5f krb5kdc[17357](info): set up 2 sockets
Jun 20 14:52:43 dist-test-slave-2m5f krb5kdc[17357](info): commencing operation
krb5kdc: starting...
W20260620 14:52:44.192337 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:52:44 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.118.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.118.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/rangeradmin_127.5.118.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/rangeradmin_127.5.118.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.118.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.118.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/rangerlookup_127.5.118.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/rangerlookup_127.5.118.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.118.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.118.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/HTTP_127.5.118.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/HTTP_127.5.118.148@KRBTEST.COM.keytab.
I20260620 14:52:44.375806 5594 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjla6OK/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/postgres -l logfile start
2026-06-20 14:52:46.946 UTC [17387] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:52:46.946 UTC [17387] LOG: listening on IPv4 address "127.5.118.148", port 38469
2026-06-20 14:52:46.950 UTC [17387] LOG: listening on Unix socket "/tmp/.s.PGSQL.38469"
2026-06-20 14:52:46.979 UTC [17390] LOG: database system was shut down at 2026-06-20 14:52:45 UTC
2026-06-20 14:52:46.982 UTC [17387] LOG: database system is ready to accept connections
I20260620 14:52:47.940027 5594 mini_postgres.cc:96] Postgres bound to 38469
2026-06-20 14:52:47.946 UTC [17398] FATAL: database "slave" does not exist
127.5.118.148:38469 - accepting connections
I20260620 14:52:47.946870 5594 mini_ranger.cc:161] Starting Ranger...
I20260620 14:52:47.962863 5594 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:52:48.464710 5594 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:52:48.464805 5594 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-admin
2026-06-20 14:52:48,761 [I] DB FLAVOR :POSTGRES
2026-06-20 14:52:48,761 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:52:48,761 [I] Checking connection..
2026-06-20 14:52:48,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:52:49,003 [I] Checking connection passed.
2026-06-20 14:52:49,003 [I] --------- Verifying version history table ---------
2026-06-20 14:52:49,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:52:49,247 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:52:49,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:52:49,519 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:52:49,519 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:52:49,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:52:49.764 UTC [17462] WARNING: there is no transaction in progress
2026-06-20 14:52:49,770 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:52:49,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:52:50,020 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:52:50,021 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:52:50,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:52:50,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:52:50,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2m5f.c.gcp-upstream.internal','N') ;"
2026-06-20 14:52:50,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:52:51,069 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:52:51,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:52:51,326 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:52:51,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:52:51,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:52:51,818 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:52:51,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:52:52.830 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:52.848 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:52.866 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:52.883 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:52.900 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:52.918 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:52.986 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:52.994 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:53.010 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:53.022 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:53.034 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:53.050 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:53.062 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:53.070 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:53.079 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:53.087 UTC [17570] WARNING: there is no transaction in progress
2026-06-20 14:52:53,816 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:52:53,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:52:54,062 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:52:54,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:52:54,320 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:52:54,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:52:54,584 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:52:54,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:52:54,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:52:55,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2m5f.c.gcp-upstream.internal';"
2026-06-20 14:52:55,335 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:52:55,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:52:55,608 [I] DB_PATCHES have already been applied
I20260620 14:52:55.614252 5594 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjla6OK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:52:55.614319 5594 mini_ranger.cc:193] Using host: 127.5.118.148
I20260620 14:52:55.617529 5594 mini_ranger.cc:227] Ranger admin URL: http://127.5.118.148:47795
Jun 20, 2026 2:52:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:52:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:52:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:52:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:52:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:52:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:52:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:52:56.757669 5594 mini_ranger.cc:160] Time spent starting Ranger: real 8.811s user 0.000s sys 0.009s
Jun 20, 2026 2:53:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:53:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 2:53:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:53:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:53:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:53:19.747642 5594 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.118.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.118.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/rangerkms_127.5.118.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.118.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/rangerkms_127.5.118.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.118.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.118.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.118.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/HTTP_127.5.118.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.118.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/HTTP_127.5.118.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:53:19 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967199, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:53:19.827024 5594 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:53:19.836750 5594 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:53:20.718243 5594 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:53:20.718346 5594 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-kms
I20260620 14:53:20.718367 5594 mini_ranger_kms.cc:227] Using postgres at 127.5.118.148:38469
2026-06-20 14:53:20,883 [I] DB FLAVOR :POSTGRES
2026-06-20 14:53:20,883 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:53:20,883 [I] Checking connection
2026-06-20 14:53:20,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:53:21,142 [I] connection success
2026-06-20 14:53:21,142 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:53:21,142 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:53:21,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:53:21,399 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:53:21,400 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:53:21,400 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:53:21,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.118.148:38469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:53:21,696 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:53:22.205440 5594 mini_ranger_kms.cc:310] Created kms service
I20260620 14:53:22.414206 5594 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:53:22.467290 5594 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:53:22 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967199, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:53:22.944278 5594 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:53:22.944403 5594 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjla6OK/build/release/bin/postgresql.jar:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjla6OK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:53:22.944429 5594 mini_ranger_kms.cc:240] Using host: 127.5.118.148
I20260620 14:53:22.948508 5594 mini_ranger_kms.cc:276] Ranger KMS PID: 17751
I20260620 14:53:22.948599 5594 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.118.148:46877
Jun 20, 2026 2:53:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:53:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:53:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:53:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:53:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:53:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 14:53:25.252588 5594 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.426s user 0.000s sys 0.148s
I20260620 14:53:25.252676 5594 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:53:25.252734 5594 mini_ranger_kms.cc:401] 127.5.118.148:46877/kms/v1/keys
Jun 20 14:53:25 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967205, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.118.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 14:53:25 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967205, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.118.148@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
Jun 20 14:53:26 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967205, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.118.148@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
Jun 20 14:53:28 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:53:28.428308 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43293
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:43293
--ranger_config_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:46877/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:28.515048 17801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:28.515300 17801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:28.515323 17801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:28.516672 17801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:28.516726 17801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:28.516739 17801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:28.516752 17801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:28.516764 17801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:28.518494 17801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.118.148:46877/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:43293
--ranger_config_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:28.518780 17801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:28.519047 17801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:28.525249 17807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:28.525249 17809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:28.525249 17806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:28.525694 17801 server_base.cc:1034] running on GCE node
I20260620 14:53:28.526059 17801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:28.526427 17801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:28.527609 17801 hybrid_clock.cc:648] HybridClock initialized: now 1781967208527584 us; error 43 us; skew 500 ppm
Jun 20 14:53:28 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967208, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:28.535115 17801 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:53:28.535738 17801 webserver.cc:460] Webserver started at http://127.5.118.190:37909/ using document root <none> and password file <none>
I20260620 14:53:28.536007 17801 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:28.536049 17801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:28.536260 17801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:53:28 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967208, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:53:28.585124 17801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "7c3f5fcb07d14c088fadca136e9b71b7"
format_stamp: "Formatted at 2026-06-20 14:53:28 on dist-test-slave-2m5f"
server_key: "8c8ee81babdf18cc02de58550170e464"
server_key_iv: "05486d5eeb720bb373aa133f0ad869cf"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:28.585547 17801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "7c3f5fcb07d14c088fadca136e9b71b7"
format_stamp: "Formatted at 2026-06-20 14:53:28 on dist-test-slave-2m5f"
server_key: "8c8ee81babdf18cc02de58550170e464"
server_key_iv: "05486d5eeb720bb373aa133f0ad869cf"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:28.587042 17801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:53:28.599665 17816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:28.599905 17801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:28.600008 17801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "7c3f5fcb07d14c088fadca136e9b71b7"
format_stamp: "Formatted at 2026-06-20 14:53:28 on dist-test-slave-2m5f"
server_key: "8c8ee81babdf18cc02de58550170e464"
server_key_iv: "05486d5eeb720bb373aa133f0ad869cf"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:28.600129 17801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:28.622798 17801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:28.625401 17801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:28.625603 17801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:28.630672 17801 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:43293
I20260620 14:53:28.630695 17868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:43293 every 8 connection(s)
I20260620 14:53:28.631110 17801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:28.632068 17869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:28.635247 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 17801
I20260620 14:53:28.635380 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:53:28.635356 17869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7: Bootstrap starting.
I20260620 14:53:28.635586 5594 external_mini_cluster.cc:1428] Setting key a6a4c23181f532e628f4727f2b5ace4e
I20260620 14:53:28.636121 17869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:28.636600 17869 log.cc:824] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:28.637452 17869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7: No bootstrap required, opened a new log
I20260620 14:53:28.640064 17869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } }
I20260620 14:53:28.640285 17869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:28.640323 17869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3f5fcb07d14c088fadca136e9b71b7, State: Initialized, Role: FOLLOWER
I20260620 14:53:28.640517 17869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } }
I20260620 14:53:28.640616 17869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:28.640647 17869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:28.640687 17869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:28.641413 17869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } }
I20260620 14:53:28.641546 17869 leader_election.cc:302] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c3f5fcb07d14c088fadca136e9b71b7; no voters:
I20260620 14:53:28.641731 17869 leader_election.cc:288] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:28.641796 17874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:28.642048 17874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 1 LEADER]: Becoming Leader. State: Replica: 7c3f5fcb07d14c088fadca136e9b71b7, State: Running, Role: LEADER
Jun 20 14:53:28 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:28.642119 17869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:28.642794 17874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } }
I20260620 14:53:28.643183 17874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c3f5fcb07d14c088fadca136e9b71b7. Latest consensus state: current_term: 1 leader_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } } }
I20260620 14:53:28.643260 17874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:28.643996 17875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } } }
I20260620 14:53:28.644084 17875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:28.644366 17869 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:53:28.644503 17882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:28.645011 17882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:28.647150 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:28.636264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.5.118.190:43293)
0620 14:53:28.636453 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:28.636462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:28.636487 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:53:28.637170 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:28.637178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:28.637233 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:28.637396 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:28.638078 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.638970 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.639772 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:28.640274 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:28.640739 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:28.643191 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:28.643195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:28.643196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:28.643209 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:28.645263 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.645726 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.645729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.645730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.645840 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:28.646080 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:28.646080 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:28.646080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:28.646315 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:28.646483 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:28.646838 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 14:53:28.646981 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260620 14:53:28.652372 17882 catalog_manager.cc:1347] Generated new cluster ID: b716c5a963ae447f847cd1a40b74876d
I20260620 14:53:28.652519 17882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:28.663848 17882 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:28.664578 17882 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:28.668810 17882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7: Generated new TSK 0
I20260620 14:53:28.669215 17882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:53:29.404027 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:46877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43293
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:29.494216 17914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:29.494468 17914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:29.494511 17914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:29.495767 17914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:29.495841 17914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:29.495886 17914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:53:29.497522 17914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.118.148:46877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:29.497818 17914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:29.498085 17914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:29.503469 17922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:29.504036 17919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:29.505232 17914 server_base.cc:1034] running on GCE node
W20260620 14:53:29.505463 17920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:29.505715 17914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:29.506047 17914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:29.507205 17914 hybrid_clock.cc:648] HybridClock initialized: now 1781967209507194 us; error 29 us; skew 500 ppm
Jun 20 14:53:29 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:29.509898 17914 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:53:29.510423 17914 webserver.cc:460] Webserver started at http://127.5.118.129:46831/ using document root <none> and password file <none>
I20260620 14:53:29.510668 17914 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:29.510712 17914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:29.510840 17914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:53:29 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:53:29.525104 17914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "45247eb039124dd6870c91d0565b3ebf"
format_stamp: "Formatted at 2026-06-20 14:53:29 on dist-test-slave-2m5f"
server_key: "e35c749223c8815779e968cd3071ddfd"
server_key_iv: "077d0490b79f3a7694d199f652739cc9"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:29.525544 17914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "45247eb039124dd6870c91d0565b3ebf"
format_stamp: "Formatted at 2026-06-20 14:53:29 on dist-test-slave-2m5f"
server_key: "e35c749223c8815779e968cd3071ddfd"
server_key_iv: "077d0490b79f3a7694d199f652739cc9"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:29.527014 17914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:53:29.541970 17929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:29.542176 17914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:29.542289 17914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "45247eb039124dd6870c91d0565b3ebf"
format_stamp: "Formatted at 2026-06-20 14:53:29 on dist-test-slave-2m5f"
server_key: "e35c749223c8815779e968cd3071ddfd"
server_key_iv: "077d0490b79f3a7694d199f652739cc9"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:29.542404 17914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:29.554205 17914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:29.556756 17914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:29.556962 17914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:29.557248 17914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:29.557642 17914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:29.557742 17914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:29.557806 17914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:29.557861 17914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:29.565243 17914 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:33743
I20260620 14:53:29.565320 18042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:33743 every 8 connection(s)
I20260620 14:53:29.565716 17914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:29.572419 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 17914
I20260620 14:53:29.572530 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:53:29.572729 5594 external_mini_cluster.cc:1428] Setting key c9765eb809e2ab7d53c342e71a5bf7d7
Jun 20 14:53:29 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:29.582124 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:29.566594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:35259 (local address 127.5.118.190:43293)
0620 14:53:29.566756 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:29.566758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:29.568509 (+ 1751us) server_negotiation.cc:407] Connection header received
0620 14:53:29.569215 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:29.569217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:29.569236 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:29.569335 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:29.570134 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:29.571041 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:29.575346 (+ 4305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:29.575671 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:29.575722 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:29.578464 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:29.578465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:29.578466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:29.578473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:29.580460 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:29.580975 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:29.580976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:29.580977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:29.581040 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:29.581343 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:29.581344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:29.581345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:29.581487 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:29.581540 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:29.581915 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 14:53:29.582007 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":104,"threads_started":1}
I20260620 14:53:29.582939 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:29.566583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43293 (local address 127.5.118.129:35259)
0620 14:53:29.568302 (+ 1719us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:29.568363 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:29.569052 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:29.569434 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:29.569439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:29.569704 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:29.569998 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:29.570005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:29.571173 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:29.571174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:29.575237 (+ 4063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:29.575241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:29.575840 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:29.575841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:29.575994 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:29.576645 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:29.576656 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:29.578347 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:29.580559 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:29.580568 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:29.580571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:29.580883 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:29.581105 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:29.581106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:29.581106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:29.581256 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:29.581575 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:29.581581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:29.581759 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 14:53:29.582513 (+ 754us) client_negotiation.cc:240] Negotiation successful
0620 14:53:29.582694 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1662,"thread_start_us":99,"threads_started":1}
I20260620 14:53:29.583775 18043 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43293
I20260620 14:53:29.583877 18043 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:29.584072 18043 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
I20260620 14:53:29.584744 17833 ts_manager.cc:194] Registered new tserver with Master: 45247eb039124dd6870c91d0565b3ebf (127.5.118.129:33743)
I20260620 14:53:29.585672 17833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:35259
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:53:29.613721 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:46877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43293
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:29.739959 18049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:29.740427 18049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:29.740527 18049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:29.742528 18049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:29.742656 18049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:29.742756 18049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:53:29.745364 18049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.118.148:46877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:29.745772 18049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:29.746094 18049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:29.752990 18058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:29.753000 18059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:29.753854 18049 server_base.cc:1034] running on GCE node
W20260620 14:53:29.754477 18061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:29.754796 18049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:29.755100 18049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:29.758394 18049 hybrid_clock.cc:648] HybridClock initialized: now 1781967209758361 us; error 45 us; skew 500 ppm
Jun 20 14:53:29 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:29.762032 18049 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:53:29.762590 18049 webserver.cc:460] Webserver started at http://127.5.118.130:42879/ using document root <none> and password file <none>
I20260620 14:53:29.762933 18049 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:29.763015 18049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:29.763211 18049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:53:29 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:53:29.781483 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "fad9e1c5c4ca428997ab9bd098d37f94"
format_stamp: "Formatted at 2026-06-20 14:53:29 on dist-test-slave-2m5f"
server_key: "17facba6649d64a42aa7f146c522ee34"
server_key_iv: "8ddc239802f52fa0cdd00cd67aea279a"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:29.782161 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "fad9e1c5c4ca428997ab9bd098d37f94"
format_stamp: "Formatted at 2026-06-20 14:53:29 on dist-test-slave-2m5f"
server_key: "17facba6649d64a42aa7f146c522ee34"
server_key_iv: "8ddc239802f52fa0cdd00cd67aea279a"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:29.784025 18049 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:53:29.797860 18068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:29.800386 18049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:53:29.800521 18049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "fad9e1c5c4ca428997ab9bd098d37f94"
format_stamp: "Formatted at 2026-06-20 14:53:29 on dist-test-slave-2m5f"
server_key: "17facba6649d64a42aa7f146c522ee34"
server_key_iv: "8ddc239802f52fa0cdd00cd67aea279a"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:29.800654 18049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 14:53:29 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:29.830755 18049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:29.843021 18049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:29.843225 18049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:29.843583 18049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:29.844058 18049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:29.844113 18049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:29.844153 18049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:29.844210 18049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:29.850984 18049 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:42163
I20260620 14:53:29.851468 18049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:29.852988 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 18049
I20260620 14:53:29.853083 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:53:29.853276 5594 external_mini_cluster.cc:1428] Setting key 3dd0e18c4eb74e8e008ddb6cef08c41e
I20260620 14:53:29.861205 18181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:42163 every 8 connection(s)
Jun 20 14:53:29 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967209, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:29.872888 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:29.853590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:45061 (local address 127.5.118.190:43293)
0620 14:53:29.853649 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:29.853651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:29.861672 (+ 8021us) server_negotiation.cc:407] Connection header received
0620 14:53:29.862443 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:29.862445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:29.862461 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:29.862493 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:29.863429 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:29.864277 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:29.865907 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:29.866223 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:29.866270 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:29.869357 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:29.869359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:29.869359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:29.869368 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:29.871118 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:29.871657 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:29.871659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:29.871659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:29.871718 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:29.872004 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:29.872005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:29.872005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:29.872147 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:29.872218 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:29.872653 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 14:53:29.872759 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:53:29.873719 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:29.852759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43293 (local address 127.5.118.130:45061)
0620 14:53:29.861487 (+ 8728us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:29.861542 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:29.862293 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:29.862711 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:29.862715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:29.862987 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:29.863284 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:29.863291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:29.864407 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:29.864408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:29.865788 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:29.865791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:29.866387 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:29.866388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:29.866537 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:29.867379 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:29.867391 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:29.869223 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:29.871224 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:29.871233 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:29.871236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:29.871563 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:29.871790 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:29.871791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:29.871791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:29.871926 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:29.872276 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:29.872284 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:29.872491 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 14:53:29.873273 (+ 782us) client_negotiation.cc:240] Negotiation successful
0620 14:53:29.873466 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8673,"thread_start_us":586,"threads_started":1}
I20260620 14:53:29.876129 18182 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43293
I20260620 14:53:29.876256 18182 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:29.876469 18182 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
I20260620 14:53:29.876972 17833 ts_manager.cc:194] Registered new tserver with Master: fad9e1c5c4ca428997ab9bd098d37f94 (127.5.118.130:42163)
I20260620 14:53:29.877499 17833 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:45061
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:53:29.899724 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:46877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43293
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:30.020339 18187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:30.020743 18187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:30.020846 18187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:30.022861 18187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:30.023000 18187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:30.023092 18187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:53:30.025772 18187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.118.148:46877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:30.026324 18187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:30.026742 18187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:30.034034 18193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:30.033964 18194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:30.034425 18187 server_base.cc:1034] running on GCE node
W20260620 14:53:30.034588 18196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:30.034898 18187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:30.035174 18187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:30.036334 18187 hybrid_clock.cc:648] HybridClock initialized: now 1781967210036311 us; error 36 us; skew 500 ppm
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:30.039738 18187 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:53:30.040333 18187 webserver.cc:460] Webserver started at http://127.5.118.131:40493/ using document root <none> and password file <none>
I20260620 14:53:30.040620 18187 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:30.040865 18187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:30.041050 18187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:53:30.062559 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "bddb679899bf4bd596740294ad962e36"
format_stamp: "Formatted at 2026-06-20 14:53:30 on dist-test-slave-2m5f"
server_key: "c0191a9930f71b9bf076a8e34ac440b6"
server_key_iv: "d25debd1a90611778ae6a91299cdf3b7"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:30.063200 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "bddb679899bf4bd596740294ad962e36"
format_stamp: "Formatted at 2026-06-20 14:53:30 on dist-test-slave-2m5f"
server_key: "c0191a9930f71b9bf076a8e34ac440b6"
server_key_iv: "d25debd1a90611778ae6a91299cdf3b7"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:30.076754 18187 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.002s
I20260620 14:53:30.094254 18203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.094520 18187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260620 14:53:30.094612 18187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "bddb679899bf4bd596740294ad962e36"
format_stamp: "Formatted at 2026-06-20 14:53:30 on dist-test-slave-2m5f"
server_key: "c0191a9930f71b9bf076a8e34ac440b6"
server_key_iv: "d25debd1a90611778ae6a91299cdf3b7"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:30.094719 18187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:30.130012 18187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:30.132823 18187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:30.133075 18187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:30.133369 18187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:30.133849 18187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:30.133955 18187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.134038 18187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:30.134174 18187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.141307 18187 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:33757
I20260620 14:53:30.141752 18316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:33757 every 8 connection(s)
I20260620 14:53:30.141930 18187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:30.144399 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 18187
I20260620 14:53:30.144502 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:53:30.144676 5594 external_mini_cluster.cc:1428] Setting key ea3330b31add31b1da5c82c960ee6a9c
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:30.154281 18045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:30.143361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:56201 (local address 127.5.118.190:43293)
0620 14:53:30.143418 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:30.143421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:30.143725 (+ 304us) server_negotiation.cc:407] Connection header received
0620 14:53:30.144527 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:30.144529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:30.144543 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:30.144575 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:30.145487 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.146221 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.147776 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.148052 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.148132 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:30.150845 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:30.150847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:30.150847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:30.150857 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:30.152618 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:30.153138 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:30.153139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:30.153139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:30.153207 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:30.153510 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:30.153511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:30.153511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:30.153662 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:30.153733 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:30.154083 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 14:53:30.154183 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:53:30.154433 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:30.143357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43293 (local address 127.5.118.131:56201)
0620 14:53:30.143553 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:30.143614 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:30.144397 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:30.144779 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:30.144783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:30.145058 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:30.145372 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:30.145379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.146378 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.146378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:30.147665 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:30.147668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.148198 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.148199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:30.148348 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:30.148973 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:30.148985 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:30.150669 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:30.152721 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:30.152729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:30.152732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:30.153050 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:30.153284 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:30.153284 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:30.153285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:30.153435 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:30.153735 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:30.153740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:30.153916 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 14:53:30.154034 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:53:30.154195 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260620 14:53:30.155311 18317 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43293
I20260620 14:53:30.155431 18317 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:30.155639 18317 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
I20260620 14:53:30.156060 17832 ts_manager.cc:194] Registered new tserver with Master: bddb679899bf4bd596740294ad962e36 (127.5.118.131:33757)
I20260620 14:53:30.156319 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:53:30.156584 17832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:56201
I20260620 14:53:30.157037 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 17914
I20260620 14:53:30.163328 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 18049
I20260620 14:53:30.168588 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 18187
I20260620 14:53:30.182005 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 17801
I20260620 14:53:30.189724 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43293
--webserver_interface=127.5.118.190
--webserver_port=37909
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:43293
--ranger_config_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:46877/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.118.148:46877/kms with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:30.273916 18320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:30.274156 18320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:30.274180 18320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:30.275525 18320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:30.275573 18320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:30.275588 18320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:30.275600 18320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:30.275612 18320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:30.277287 18320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.118.148:46877/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:43293
--ranger_config_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:43293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=37909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:30.277557 18320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:30.277808 18320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:30.283139 18328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:30.283224 18326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:30.283172 18320 server_base.cc:1034] running on GCE node
W20260620 14:53:30.283139 18325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:30.283565 18320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:30.283829 18320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:30.284993 18320 hybrid_clock.cc:648] HybridClock initialized: now 1781967210284974 us; error 30 us; skew 500 ppm
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:30.288547 18320 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:53:30.289211 18320 webserver.cc:460] Webserver started at http://127.5.118.190:37909/ using document root <none> and password file <none>
I20260620 14:53:30.289465 18320 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:30.289551 18320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:53:30.376147 18320 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:30.377249 18336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.377476 18320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:53:30.377589 18320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "7c3f5fcb07d14c088fadca136e9b71b7"
format_stamp: "Formatted at 2026-06-20 14:53:28 on dist-test-slave-2m5f"
server_key: "8c8ee81babdf18cc02de58550170e464"
server_key_iv: "05486d5eeb720bb373aa133f0ad869cf"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:30.378087 18320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:30.392253 18320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:30.394492 18320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:30.394667 18320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:30.399078 18320 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:43293
I20260620 14:53:30.399114 18388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:43293 every 8 connection(s)
I20260620 14:53:30.399560 18320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:30.400965 18389 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:53:30.401628 18389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7: Bootstrap starting.
I20260620 14:53:30.407004 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 18320
I20260620 14:53:30.407361 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:46877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:33743
--local_ip_for_outbound_sockets=127.5.118.129
--tserver_master_addrs=127.5.118.190:43293
--webserver_port=46831
--webserver_interface=127.5.118.129
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.118.148:46877/kms with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:30.410266 18389 log.cc:824] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:30.412376 18389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:53:30.412720 18389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7: Bootstrap complete.
I20260620 14:53:30.415202 18389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } }
I20260620 14:53:30.415447 18389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3f5fcb07d14c088fadca136e9b71b7, State: Initialized, Role: FOLLOWER
I20260620 14:53:30.415628 18389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } }
I20260620 14:53:30.415716 18389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:30.415768 18389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:30.415818 18389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:53:30.417004 18389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } }
I20260620 14:53:30.417096 18389 leader_election.cc:302] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c3f5fcb07d14c088fadca136e9b71b7; no voters:
I20260620 14:53:30.417320 18389 leader_election.cc:288] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:53:30.417356 18393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:53:30.417464 18393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [term 2 LEADER]: Becoming Leader. State: Replica: 7c3f5fcb07d14c088fadca136e9b71b7, State: Running, Role: LEADER
I20260620 14:53:30.417554 18393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } }
I20260620 14:53:30.417600 18389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:30.417822 18394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } } }
I20260620 14:53:30.418018 18394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:30.417965 18395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c3f5fcb07d14c088fadca136e9b71b7. Latest consensus state: current_term: 2 leader_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3f5fcb07d14c088fadca136e9b71b7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 43293 } } }
I20260620 14:53:30.418249 18395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:30.418329 18400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:30.418859 18400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:30.418996 18400 catalog_manager.cc:1259] Loaded cluster ID: b716c5a963ae447f847cd1a40b74876d
I20260620 14:53:30.419039 18400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:30.419212 18389 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:53:30.422693 18400 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:30.423344 18400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7: Loaded TSK: 0
I20260620 14:53:30.423689 18400 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:53:30.498543 18391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:30.498955 18391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:30.499055 18391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:30.500996 18391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:30.501130 18391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:30.501223 18391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:53:30.503362 18391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.118.148:46877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:33743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=46831
--tserver_master_addrs=127.5.118.190:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:30.503695 18391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:30.504002 18391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:30.509586 18427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:30.509586 18429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:30.510047 18426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:30.510313 18391 server_base.cc:1034] running on GCE node
I20260620 14:53:30.510576 18391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:30.510838 18391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:30.514094 18391 hybrid_clock.cc:648] HybridClock initialized: now 1781967210514068 us; error 39 us; skew 500 ppm
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:30.516991 18391 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:53:30.517522 18391 webserver.cc:460] Webserver started at http://127.5.118.129:46831/ using document root <none> and password file <none>
I20260620 14:53:30.517985 18391 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:30.518107 18391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:53:30.546291 18391 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:53:30.547363 18437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.547556 18391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:30.547658 18391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "45247eb039124dd6870c91d0565b3ebf"
format_stamp: "Formatted at 2026-06-20 14:53:29 on dist-test-slave-2m5f"
server_key: "e35c749223c8815779e968cd3071ddfd"
server_key_iv: "077d0490b79f3a7694d199f652739cc9"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:30.548233 18391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:30.569521 18391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:30.572037 18391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:30.572270 18391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:30.572494 18391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:30.572876 18391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:30.572925 18391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.572963 18391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:30.572989 18391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.579576 18391 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:33743
I20260620 14:53:30.580075 18391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:30.586449 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 18391
I20260620 14:53:30.586767 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:46877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:42163
--local_ip_for_outbound_sockets=127.5.118.130
--tserver_master_addrs=127.5.118.190:43293
--webserver_port=42879
--webserver_interface=127.5.118.130
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.118.148:46877/kms with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:30.592957 18550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:33743 every 8 connection(s)
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:30.619598 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:30.594134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:49435 (local address 127.5.118.190:43293)
0620 14:53:30.594811 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:30.594822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:30.595761 (+ 939us) server_negotiation.cc:407] Connection header received
0620 14:53:30.595929 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:30.595935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:30.595994 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:30.596141 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:30.597011 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.598474 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.599950 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.600620 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.600725 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:30.604525 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:30.604531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:30.604531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:30.604546 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:30.616773 (+ 12227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:30.617486 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:30.617490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:30.617490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:30.617616 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:30.618014 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:30.618016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:30.618016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:30.618397 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:30.618483 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:30.619256 (+ 773us) server_negotiation.cc:299] Negotiation successful
0620 14:53:30.619382 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":90,"threads_started":1}
I20260620 14:53:30.619915 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:30.593728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43293 (local address 127.5.118.129:49435)
0620 14:53:30.594915 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:30.594967 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:30.595587 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:30.596353 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:30.596358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:30.596602 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:30.596877 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:30.596883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.598603 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.598604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:30.599849 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:30.599852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.600844 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.600845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:30.600981 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:30.602610 (+ 1629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:30.602622 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:30.604358 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:30.616972 (+ 12614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:30.616987 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:30.616990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:30.617365 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:30.617748 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:30.617749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:30.617749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:30.617901 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:30.618514 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:30.618522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:30.618699 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 14:53:30.618861 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 14:53:30.619079 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"thread_start_us":107,"threads_started":1}
I20260620 14:53:30.620853 18551 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43293
I20260620 14:53:30.620970 18551 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:30.621181 18551 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
W20260620 14:53:30.622176 18551 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:43293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:53:30.622270 18551 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:30.622349 18551 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
I20260620 14:53:30.622658 18551 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:30.622741 18551 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
W20260620 14:53:30.623027 18551 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:53:30.719908 18553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:30.720148 18553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:30.720216 18553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:30.721442 18553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:30.721511 18553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:30.721565 18553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:53:30.723057 18553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.118.148:46877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:42163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=42879
--tserver_master_addrs=127.5.118.190:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:30.723330 18553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:30.723604 18553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:30.728982 18563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:30.729254 18560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:30.729722 18561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:30.730525 18553 server_base.cc:1034] running on GCE node
I20260620 14:53:30.730785 18553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:30.731202 18553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:30.732374 18553 hybrid_clock.cc:648] HybridClock initialized: now 1781967210732351 us; error 38 us; skew 500 ppm
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:30.735366 18553 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:53:30.735904 18553 webserver.cc:460] Webserver started at http://127.5.118.130:42879/ using document root <none> and password file <none>
I20260620 14:53:30.736246 18553 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:30.736336 18553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:53:30.760630 18553 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 14:53:30.761675 18571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.761870 18553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:30.761972 18553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "fad9e1c5c4ca428997ab9bd098d37f94"
format_stamp: "Formatted at 2026-06-20 14:53:29 on dist-test-slave-2m5f"
server_key: "17facba6649d64a42aa7f146c522ee34"
server_key_iv: "8ddc239802f52fa0cdd00cd67aea279a"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:30.762396 18553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:30.789794 18553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:30.792378 18553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:30.792567 18553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:30.792865 18553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:30.793263 18553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:30.793320 18553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.793362 18553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:30.793390 18553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.801014 18553 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:42163
I20260620 14:53:30.801470 18553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:30.802029 18684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:42163 every 8 connection(s)
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:30.808697 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 18553
I20260620 14:53:30.809000 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.118.148:46877/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:33757
--local_ip_for_outbound_sockets=127.5.118.131
--tserver_master_addrs=127.5.118.190:43293
--webserver_port=40493
--webserver_interface=127.5.118.131
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:40799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.118.148:46877/kms with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:30.813249 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:30.802674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:43037 (local address 127.5.118.190:43293)
0620 14:53:30.802717 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:30.802719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:30.802730 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:53:30.803409 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:30.803412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:30.803430 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:30.803508 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:30.804144 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.804971 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.806210 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.806441 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.806527 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:30.809577 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:30.809579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:30.809580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:30.809590 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:30.811464 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:30.812028 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:30.812029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:30.812030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:30.812093 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:30.812436 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:30.812437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:30.812437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:30.812595 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:30.812640 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:30.813037 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 14:53:30.813143 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:53:30.814083 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:30.802338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43293 (local address 127.5.118.130:43037)
0620 14:53:30.802518 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:30.802564 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:30.803262 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:30.803581 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:30.803584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:30.803818 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:30.804033 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:30.804038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.805102 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.805104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:30.806094 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:30.806097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:30.806560 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:30.806561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:30.806745 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:30.807423 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:30.807433 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:30.809427 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:30.811586 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:30.811596 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:30.811599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:30.811930 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:30.812167 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:30.812168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:30.812197 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:30.812352 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:30.812689 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:30.812695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:30.812877 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 14:53:30.813645 (+ 768us) client_negotiation.cc:240] Negotiation successful
0620 14:53:30.813835 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260620 14:53:30.814751 18685 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43293
I20260620 14:53:30.814899 18685 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:30.815155 18685 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
W20260620 14:53:30.815901 18685 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:43293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:53:30.816032 18685 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:30.816138 18685 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
I20260620 14:53:30.816514 18685 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:30.816632 18685 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
W20260620 14:53:30.816952 18685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:53:30.924007 18688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:30.924319 18688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:30.924369 18688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:30.925766 18688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:30.925836 18688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:30.925881 18688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:53:30.927402 18688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:40799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.118.148:46877/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:33757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=40493
--tserver_master_addrs=127.5.118.190:43293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:30.927701 18688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:30.927942 18688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:30.934149 18696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:30.934409 18694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:30.934725 18693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:30.935005 18688 server_base.cc:1034] running on GCE node
I20260620 14:53:30.935324 18688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:30.935760 18688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:30.936924 18688 hybrid_clock.cc:648] HybridClock initialized: now 1781967210936913 us; error 30 us; skew 500 ppm
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:30.939630 18688 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:53:30.940109 18688 webserver.cc:460] Webserver started at http://127.5.118.131:40493/ using document root <none> and password file <none>
I20260620 14:53:30.940440 18688 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:30.940529 18688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:53:30 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:53:30.959156 18688 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:53:30.960280 18704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.960597 18688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:53:30.960716 18688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "bddb679899bf4bd596740294ad962e36"
format_stamp: "Formatted at 2026-06-20 14:53:30 on dist-test-slave-2m5f"
server_key: "c0191a9930f71b9bf076a8e34ac440b6"
server_key_iv: "d25debd1a90611778ae6a91299cdf3b7"
server_key_version: "kuduclusterkey@0"
I20260620 14:53:30.961185 18688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:30.993148 18688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:30.995834 18688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:30.996024 18688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:30.996392 18688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:30.996824 18688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:30.996879 18688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:30.996919 18688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:30.996943 18688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:31.003683 18688 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:33757
I20260620 14:53:31.005879 18818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:33757 every 8 connection(s)
I20260620 14:53:31.006069 18688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:31.011353 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 18688
Jun 20 14:53:31 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967210, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:31.017364 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:31.006949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:43293 (local address 127.5.118.131:50743)
0620 14:53:31.007340 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:31.007374 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:31.007896 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:31.008230 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:31.008235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:31.008426 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:31.008637 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:31.008642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.009361 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.009363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:31.010238 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:31.010240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.010690 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.010691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:31.010816 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:31.011520 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:31.011533 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:31.013279 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:31.015380 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:31.015398 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:31.015400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:31.015768 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:31.016019 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:31.016020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:31.016020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:31.016240 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:31.016566 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:31.016573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:31.016779 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 14:53:31.016890 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:53:31.017076 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":208,"threads_started":1}
I20260620 14:53:31.017445 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:31.006923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:50743 (local address 127.5.118.190:43293)
0620 14:53:31.006989 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:31.006991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:31.007482 (+ 491us) server_negotiation.cc:407] Connection header received
0620 14:53:31.008033 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:31.008035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:31.008049 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:31.008096 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:31.008741 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.009254 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.010337 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.010561 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.010654 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:31.013391 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:31.013393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:31.013393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:31.013401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:31.015285 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.015862 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.015863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.015864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.015933 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.016323 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.016325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.016325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.016479 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:31.016557 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:31.016948 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 14:53:31.017339 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:53:31.017823 18819 heartbeater.cc:347] Connected to a master server at 127.5.118.190:43293
I20260620 14:53:31.017938 18819 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:31.018173 18819 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
W20260620 14:53:31.018899 18819 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:43293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:53:31.018975 18819 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:31.019124 18819 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
I20260620 14:53:31.019503 18819 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:31.019587 18819 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
W20260620 14:53:31.019897 18819 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:53:31.020481 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:31.011826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.5.118.190:43293)
0620 14:53:31.011950 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:31.011951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:31.012104 (+ 153us) server_negotiation.cc:407] Connection header received
0620 14:53:31.012160 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:31.012161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:31.012215 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:31.012297 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:31.012941 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.013709 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.014860 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:31.015183 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:31.015286 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:31.016703 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:31.016704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:31.016704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:31.016711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:31.019115 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.019623 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.019669 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.019669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.019734 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:31.019925 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:31.019926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:31.019926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:31.020049 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:31.020226 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:31.020292 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:53:31.020378 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
Jun 20 14:53:31 dist-test-slave-2m5f krb5kdc[17357](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967211, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:31.623224 18551 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:31.623396 18551 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
I20260620 14:53:31.623895 18353 ts_manager.cc:194] Registered new tserver with Master: 45247eb039124dd6870c91d0565b3ebf (127.5.118.129:33743)
I20260620 14:53:31.624931 18353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:49435
I20260620 14:53:31.817664 18685 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:31.817839 18685 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
I20260620 14:53:31.818400 18352 ts_manager.cc:194] Registered new tserver with Master: fad9e1c5c4ca428997ab9bd098d37f94 (127.5.118.130:42163)
I20260620 14:53:31.818984 18352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:43037
Jun 20 14:53:31 dist-test-slave-2m5f krb5kdc[17357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967211, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for HTTP/127.5.118.148@KRBTEST.COM
I20260620 14:53:32.020102 18819 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:32.020295 18819 heartbeater.cc:511] Master 127.5.118.190:43293 requested a full tablet report, sending...
I20260620 14:53:32.020637 18353 ts_manager.cc:194] Registered new tserver with Master: bddb679899bf4bd596740294ad962e36 (127.5.118.131:33757)
I20260620 14:53:32.021168 18353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:50743
I20260620 14:53:32.022608 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:53:32.032442 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.023504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.5.118.190:43293)
0620 14:53:32.023779 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.023782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.023934 (+ 152us) server_negotiation.cc:407] Connection header received
0620 14:53:32.023956 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.023956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.023967 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.024022 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:32.024782 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.025553 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.026753 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.027077 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.027136 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.028801 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:32.028803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:32.028804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:32.028812 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:32.030902 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:32.031301 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:32.031302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:32.031303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:32.031360 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:32.031534 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:32.031535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:32.031535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:32.031677 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:32.031768 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.032242 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.032328 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":73,"threads_started":1}
I20260620 14:53:32.034718 18353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:53:32.037078 18353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:32.045686 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.041948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:33743 (local address 127.0.0.1:56374)
0620 14:53:32.042147 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.042198 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.042273 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.042481 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.042484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.042485 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:32.042565 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.042568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.043525 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.043526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.044445 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.044448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.045355 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.045356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.045440 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.045455 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.045529 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.045591 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
I20260620 14:53:32.045686 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.042041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.5.118.129:33743)
0620 14:53:32.042173 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.042178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.042251 (+ 73us) server_negotiation.cc:407] Connection header received
0620 14:53:32.042361 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.042364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.042379 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.042476 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:32.042653 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.043421 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.044634 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.045260 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.045354 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.045420 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.045558 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.045598 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":44,"threads_started":1}
I20260620 14:53:32.046914 18474 tablet_service.cc:1467] Processing CreateTablet for tablet 8d049ca2fa8840d5a63601be707c85fa (DEFAULT_TABLE table=test-table [id=6eefde8d234f4c2ba1c675c284abf114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:32.047358 18474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d049ca2fa8840d5a63601be707c85fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:32.051074 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.045798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:42163 (local address 127.0.0.1:37848)
0620 14:53:32.046246 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.046257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.046293 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.047119 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.047120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.047122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:32.047156 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.047157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.048394 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.048395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.049461 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.049464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.050817 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.050818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.050881 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.050885 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.050934 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.050988 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":92,"threads_started":1}
I20260620 14:53:32.051772 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.045927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.5.118.130:42163)
0620 14:53:32.046614 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.046618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.046627 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:53:32.046671 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.046673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.046690 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.046773 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:32.047246 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.048298 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.049565 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.050638 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.050722 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.050782 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.051646 (+ 864us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.051698 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":71,"threads_started":1}
I20260620 14:53:32.052426 18619 tablet_service.cc:1467] Processing CreateTablet for tablet 8d049ca2fa8840d5a63601be707c85fa (DEFAULT_TABLE table=test-table [id=6eefde8d234f4c2ba1c675c284abf114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:32.052800 18619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d049ca2fa8840d5a63601be707c85fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:32.055437 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.045997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:33757 (local address 127.0.0.1:44450)
0620 14:53:32.051153 (+ 5156us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.051166 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.051212 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.051508 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.051509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.051510 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:32.051533 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.051534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.053069 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.053069 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.054163 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.054166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.055193 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.055194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.055262 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.055267 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.055321 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.055363 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5144,"thread_start_us":62,"threads_started":1}
I20260620 14:53:32.055816 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.046149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.5.118.131:33757)
0620 14:53:32.046542 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.046546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.051283 (+ 4737us) server_negotiation.cc:407] Connection header received
0620 14:53:32.051324 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.051326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.051342 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.051407 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:32.051612 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.052973 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.054275 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.055010 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.055087 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.055145 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.055670 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.055721 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":60,"threads_started":1}
I20260620 14:53:32.056733 18753 tablet_service.cc:1467] Processing CreateTablet for tablet 8d049ca2fa8840d5a63601be707c85fa (DEFAULT_TABLE table=test-table [id=6eefde8d234f4c2ba1c675c284abf114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:32.057072 18753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d049ca2fa8840d5a63601be707c85fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:32.059780 18854 tablet_bootstrap.cc:492] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36: Bootstrap starting.
I20260620 14:53:32.060359 18853 tablet_bootstrap.cc:492] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf: Bootstrap starting.
I20260620 14:53:32.060631 18854 tablet_bootstrap.cc:654] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:32.060812 18855 tablet_bootstrap.cc:492] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94: Bootstrap starting.
I20260620 14:53:32.061064 18853 tablet_bootstrap.cc:654] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:32.061088 18854 log.cc:824] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:32.061393 18853 log.cc:824] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:32.061458 18855 tablet_bootstrap.cc:654] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:32.061765 18855 log.cc:824] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:32.062099 18854 tablet_bootstrap.cc:492] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36: No bootstrap required, opened a new log
I20260620 14:53:32.062155 18854 ts_tablet_manager.cc:1397] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:32.064123 18854 raft_consensus.cc:348] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45247eb039124dd6870c91d0565b3ebf" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 33743 } } peers { permanent_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 42163 } } peers { permanent_uuid: "bddb679899bf4bd596740294ad962e36" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33757 } }
I20260620 14:53:32.064329 18854 raft_consensus.cc:374] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:32.064371 18854 raft_consensus.cc:729] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bddb679899bf4bd596740294ad962e36, State: Initialized, Role: FOLLOWER
I20260620 14:53:32.064498 18854 consensus_queue.cc:260] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45247eb039124dd6870c91d0565b3ebf" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 33743 } } peers { permanent_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 42163 } } peers { permanent_uuid: "bddb679899bf4bd596740294ad962e36" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33757 } }
I20260620 14:53:32.064776 18854 ts_tablet_manager.cc:1428] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:32.064812 18819 heartbeater.cc:503] Master 127.5.118.190:43293 was elected leader, sending a full tablet report...
I20260620 14:53:32.065752 18855 tablet_bootstrap.cc:492] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94: No bootstrap required, opened a new log
I20260620 14:53:32.065894 18855 ts_tablet_manager.cc:1397] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 14:53:32.067760 18855 raft_consensus.cc:348] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45247eb039124dd6870c91d0565b3ebf" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 33743 } } peers { permanent_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 42163 } } peers { permanent_uuid: "bddb679899bf4bd596740294ad962e36" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33757 } }
I20260620 14:53:32.067930 18855 raft_consensus.cc:374] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:32.067973 18855 raft_consensus.cc:729] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad9e1c5c4ca428997ab9bd098d37f94, State: Initialized, Role: FOLLOWER
I20260620 14:53:32.068099 18855 consensus_queue.cc:260] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45247eb039124dd6870c91d0565b3ebf" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 33743 } } peers { permanent_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 42163 } } peers { permanent_uuid: "bddb679899bf4bd596740294ad962e36" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33757 } }
I20260620 14:53:32.068434 18685 heartbeater.cc:503] Master 127.5.118.190:43293 was elected leader, sending a full tablet report...
I20260620 14:53:32.068431 18855 ts_tablet_manager.cc:1428] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:32.068871 18853 tablet_bootstrap.cc:492] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf: No bootstrap required, opened a new log
I20260620 14:53:32.068971 18853 ts_tablet_manager.cc:1397] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260620 14:53:32.071637 18853 raft_consensus.cc:348] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45247eb039124dd6870c91d0565b3ebf" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 33743 } } peers { permanent_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 42163 } } peers { permanent_uuid: "bddb679899bf4bd596740294ad962e36" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33757 } }
I20260620 14:53:32.071856 18853 raft_consensus.cc:374] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:32.071898 18853 raft_consensus.cc:729] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45247eb039124dd6870c91d0565b3ebf, State: Initialized, Role: FOLLOWER
I20260620 14:53:32.072036 18853 consensus_queue.cc:260] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45247eb039124dd6870c91d0565b3ebf" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 33743 } } peers { permanent_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 42163 } } peers { permanent_uuid: "bddb679899bf4bd596740294ad962e36" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33757 } }
I20260620 14:53:32.072412 18853 ts_tablet_manager.cc:1428] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 14:53:32.072430 18551 heartbeater.cc:503] Master 127.5.118.190:43293 was elected leader, sending a full tablet report...
W20260620 14:53:32.094463 18552 tablet.cc:2369] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:32.116806 18861 raft_consensus.cc:482] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:32.116973 18861 raft_consensus.cc:504] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45247eb039124dd6870c91d0565b3ebf" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 33743 } } peers { permanent_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 42163 } } peers { permanent_uuid: "bddb679899bf4bd596740294ad962e36" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33757 } }
I20260620 14:53:32.117393 18861 leader_election.cc:288] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fad9e1c5c4ca428997ab9bd098d37f94 (127.5.118.130:42163), bddb679899bf4bd596740294ad962e36 (127.5.118.131:33757)
I20260620 14:53:32.121954 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.117955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:47361 (local address 127.5.118.131:33757)
0620 14:53:32.118000 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.118002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.118126 (+ 124us) server_negotiation.cc:407] Connection header received
0620 14:53:32.118226 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.118227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.118236 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.118263 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:32.118482 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.119262 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.120939 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.121514 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.121564 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.121586 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.121831 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.121877 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:53:32.122263 18863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.117631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:33757 (local address 127.5.118.129:47361)
0620 14:53:32.118061 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.118072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.118153 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.118332 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.118333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.118335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:32.118412 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.118415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.119393 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.119394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.120817 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.120821 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.121665 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.121666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.121738 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.121742 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.122113 (+ 371us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.122163 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":83,"thread_start_us":150,"threads_started":1}
I20260620 14:53:32.122604 18773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d049ca2fa8840d5a63601be707c85fa" candidate_uuid: "45247eb039124dd6870c91d0565b3ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bddb679899bf4bd596740294ad962e36" is_pre_election: true
I20260620 14:53:32.122774 18773 raft_consensus.cc:2454] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45247eb039124dd6870c91d0565b3ebf in term 0.
I20260620 14:53:32.122967 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.117872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:54983 (local address 127.5.118.130:42163)
0620 14:53:32.117920 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.117922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.118126 (+ 204us) server_negotiation.cc:407] Connection header received
0620 14:53:32.118233 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.118234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.118244 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.118274 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:32.118505 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.119242 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.120937 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.121526 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.121579 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.121588 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.122840 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.122894 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:53:32.123061 18438 leader_election.cc:302] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45247eb039124dd6870c91d0565b3ebf, bddb679899bf4bd596740294ad962e36; no voters:
I20260620 14:53:32.123198 18861 raft_consensus.cc:2781] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:32.123243 18861 raft_consensus.cc:482] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:32.123265 18861 raft_consensus.cc:3037] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:32.123519 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.117592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:42163 (local address 127.5.118.129:54983)
0620 14:53:32.118054 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.118065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:32.118154 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:32.118341 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:32.118342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:32.118344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:32.118425 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.118427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.119371 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.119372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.120821 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:32.120825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.122539 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.122540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:32.122621 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.122626 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:53:32.123373 (+ 747us) client_negotiation.cc:240] Negotiation successful
0620 14:53:32.123426 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":63,"thread_start_us":117,"threads_started":1}
I20260620 14:53:32.124023 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d049ca2fa8840d5a63601be707c85fa" candidate_uuid: "45247eb039124dd6870c91d0565b3ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" is_pre_election: true
I20260620 14:53:32.124152 18861 raft_consensus.cc:504] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45247eb039124dd6870c91d0565b3ebf" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 33743 } } peers { permanent_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 42163 } } peers { permanent_uuid: "bddb679899bf4bd596740294ad962e36" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33757 } }
I20260620 14:53:32.124248 18623 raft_consensus.cc:2454] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45247eb039124dd6870c91d0565b3ebf in term 0.
I20260620 14:53:32.124492 18623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d049ca2fa8840d5a63601be707c85fa" candidate_uuid: "45247eb039124dd6870c91d0565b3ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad9e1c5c4ca428997ab9bd098d37f94"
I20260620 14:53:32.124521 18773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d049ca2fa8840d5a63601be707c85fa" candidate_uuid: "45247eb039124dd6870c91d0565b3ebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bddb679899bf4bd596740294ad962e36"
I20260620 14:53:32.124572 18623 raft_consensus.cc:3037] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:32.124617 18773 raft_consensus.cc:3037] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:32.125311 18623 raft_consensus.cc:2454] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45247eb039124dd6870c91d0565b3ebf in term 1.
I20260620 14:53:32.125464 18773 raft_consensus.cc:2454] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45247eb039124dd6870c91d0565b3ebf in term 1.
I20260620 14:53:32.125500 18439 leader_election.cc:302] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45247eb039124dd6870c91d0565b3ebf, fad9e1c5c4ca428997ab9bd098d37f94; no voters:
I20260620 14:53:32.125697 18861 leader_election.cc:288] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [CANDIDATE]: Term 1 election: Requested vote from peers fad9e1c5c4ca428997ab9bd098d37f94 (127.5.118.130:42163), bddb679899bf4bd596740294ad962e36 (127.5.118.131:33757)
I20260620 14:53:32.125770 18861 raft_consensus.cc:2781] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:32.125828 18861 raft_consensus.cc:686] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [term 1 LEADER]: Becoming Leader. State: Replica: 45247eb039124dd6870c91d0565b3ebf, State: Running, Role: LEADER
I20260620 14:53:32.125943 18861 consensus_queue.cc:237] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45247eb039124dd6870c91d0565b3ebf" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 33743 } } peers { permanent_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 42163 } } peers { permanent_uuid: "bddb679899bf4bd596740294ad962e36" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33757 } }
I20260620 14:53:32.126833 18353 catalog_manager.cc:5495] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf reported cstate change: term changed from 0 to 1, leader changed from <none> to 45247eb039124dd6870c91d0565b3ebf (127.5.118.129). New cstate: current_term: 1 leader_uuid: "45247eb039124dd6870c91d0565b3ebf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45247eb039124dd6870c91d0565b3ebf" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 33743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bddb679899bf4bd596740294ad962e36" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33757 } health_report { overall_health: UNKNOWN } } }
I20260620 14:53:32.146924 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.143444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.5.118.129:33743)
0620 14:53:32.143487 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:32.143489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:32.143499 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:53:32.143573 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:32.143574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:32.143596 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:32.143688 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:32.143855 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.144643 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.145795 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:32.146109 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:32.146171 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:32.146384 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:32.146532 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:32.146610 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:32.146808 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 14:53:32.146832 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:53:32.149386 18773 raft_consensus.cc:1261] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36 [term 1 FOLLOWER]: Refusing update from remote peer 45247eb039124dd6870c91d0565b3ebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:32.149463 18623 raft_consensus.cc:1261] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94 [term 1 FOLLOWER]: Refusing update from remote peer 45247eb039124dd6870c91d0565b3ebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:32.149811 18864 consensus_queue.cc:1035] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fad9e1c5c4ca428997ab9bd098d37f94" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:32.150015 18864 consensus_queue.cc:1035] T 8d049ca2fa8840d5a63601be707c85fa P 45247eb039124dd6870c91d0565b3ebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bddb679899bf4bd596740294ad962e36" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:32.153311 18867 mvcc.cc:205] Tried to move back new op lower bound from 7298937700961173504 to 7298937700868595712. Current Snapshot: MvccSnapshot[applied={T|T < 7298937700961173504}]
I20260620 14:53:32.153508 18868 mvcc.cc:205] Tried to move back new op lower bound from 7298937700961173504 to 7298937700868595712. Current Snapshot: MvccSnapshot[applied={T|T < 7298937700961173504}]
I20260620 14:53:32.155383 18869 mvcc.cc:205] Tried to move back new op lower bound from 7298937700961173504 to 7298937700868595712. Current Snapshot: MvccSnapshot[applied={T|T < 7298937700961173504}]
I20260620 14:53:32.157552 18352 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:32.157675 18352 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:32.158577 18352 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c3f5fcb07d14c088fadca136e9b71b7: Sending DeleteTablet for 3 replicas of tablet 8d049ca2fa8840d5a63601be707c85fa
I20260620 14:53:32.158969 18619 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d049ca2fa8840d5a63601be707c85fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:32 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37848
I20260620 14:53:32.158988 18753 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d049ca2fa8840d5a63601be707c85fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:32 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:44450
I20260620 14:53:32.159425 18474 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d049ca2fa8840d5a63601be707c85fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:32 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:56374
I20260620 14:53:32.159709 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 18391
I20260620 14:53:32.160562 18876 tablet_replica.cc:330] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36: stopping tablet replica
I20260620 14:53:32.160686 18876 raft_consensus.cc:2229] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:32.160795 18876 raft_consensus.cc:2258] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:32.160959 18877 tablet_replica.cc:330] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94: stopping tablet replica
I20260620 14:53:32.161058 18877 raft_consensus.cc:2229] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:32.161160 18877 raft_consensus.cc:2258] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:32.162142 18876 ts_tablet_manager.cc:1905] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:32.163769 18876 ts_tablet_manager.cc:1918] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:53:32.163841 18876 log.cc:1196] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/8d049ca2fa8840d5a63601be707c85fa
I20260620 14:53:32.164219 18876 ts_tablet_manager.cc:1939] T 8d049ca2fa8840d5a63601be707c85fa P bddb679899bf4bd596740294ad962e36: Deleting consensus metadata
I20260620 14:53:32.164739 18337 catalog_manager.cc:4841] TS bddb679899bf4bd596740294ad962e36 (127.5.118.131:33757): tablet 8d049ca2fa8840d5a63601be707c85fa (table test-table [id=6eefde8d234f4c2ba1c675c284abf114]) successfully deleted
W20260620 14:53:32.166980 18339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:33743 (error 108)
I20260620 14:53:32.167713 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 18553
I20260620 14:53:32.168118 18877 ts_tablet_manager.cc:1905] T 8d049ca2fa8840d5a63601be707c85fa P fad9e1c5c4ca428997ab9bd098d37f94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:32.172721 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.172545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:33743 (local address 127.0.0.1:56390)
0620 14:53:32.172599 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.172673 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:33743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 14:53:32.172987 18339 catalog_manager.cc:4584] TS 45247eb039124dd6870c91d0565b3ebf (127.5.118.129:33743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d049ca2fa8840d5a63601be707c85fa: Network error: Client connection negotiation failed: client connection to 127.5.118.129:33743: connect: Connection refused (error 111)
I20260620 14:53:32.177106 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:32.176995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:42163 (local address 127.0.0.1:37858)
0620 14:53:32.177039 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:32.177063 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.130:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:53:32.177198 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 18688
I20260620 14:53:32.183173 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 18320
I20260620 14:53:32.189446 5594 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:53:32.524453 5594 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:53:32.524544 5594 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:53:32.869947 5594 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:53:32.870 UTC [17387] LOG: received smart shutdown request
2026-06-20 14:53:32.874 UTC [17387] LOG: background worker "logical replication launcher" (PID 17396) exited with exit code 1
2026-06-20 14:53:32.874 UTC [17391] LOG: shutting down
2026-06-20 14:53:33.064 UTC [17387] LOG: database system is shut down
2026-06-20T14:53:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50040 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:33 dist-test-slave-2m5f krb5kdc[18888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:33 dist-test-slave-2m5f krb5kdc[18888](info): set up 2 sockets
Jun 20 14:53:33 dist-test-slave-2m5f krb5kdc[18888](info): commencing operation
krb5kdc: starting...
W20260620 14:53:34.236163 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:53:34.367574 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:44269
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:44269,127.5.118.189:35373,127.5.118.188:43341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:34.459836 18901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:34.460096 18901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:34.460124 18901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:34.461612 18901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:34.461674 18901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:34.461692 18901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:34.461704 18901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:34.461719 18901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:34.463485 18901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:44269,127.5.118.189:35373,127.5.118.188:43341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:44269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:34.463775 18901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:34.464048 18901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:34.469924 18906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:34.470074 18901 server_base.cc:1034] running on GCE node
W20260620 14:53:34.469897 18909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:34.469915 18907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:34.470422 18901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:34.470784 18901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:34.472070 18901 hybrid_clock.cc:648] HybridClock initialized: now 1781967214472034 us; error 76 us; skew 500 ppm
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:34.480674 18901 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:53:34.481209 18901 webserver.cc:460] Webserver started at http://127.5.118.190:42153/ using document root <none> and password file <none>
I20260620 14:53:34.481467 18901 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:34.481518 18901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:34.481657 18901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:34.482856 18901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "e63e9f5bbbe742e9a2764d84513f2457"
format_stamp: "Formatted at 2026-06-20 14:53:34 on dist-test-slave-2m5f"
server_key: "af8ab34d1f52801443ae2818a6e779b8"
server_key_iv: "38291cd900c7a0fa17f2770539437321"
server_key_version: "encryptionkey@0"
I20260620 14:53:34.483242 18901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "e63e9f5bbbe742e9a2764d84513f2457"
format_stamp: "Formatted at 2026-06-20 14:53:34 on dist-test-slave-2m5f"
server_key: "af8ab34d1f52801443ae2818a6e779b8"
server_key_iv: "38291cd900c7a0fa17f2770539437321"
server_key_version: "encryptionkey@0"
I20260620 14:53:34.484740 18901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:53:34.485656 18916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:34.485908 18901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:53:34.486039 18901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "e63e9f5bbbe742e9a2764d84513f2457"
format_stamp: "Formatted at 2026-06-20 14:53:34 on dist-test-slave-2m5f"
server_key: "af8ab34d1f52801443ae2818a6e779b8"
server_key_iv: "38291cd900c7a0fa17f2770539437321"
server_key_version: "encryptionkey@0"
I20260620 14:53:34.486121 18901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:34.506790 18901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:34.509212 18901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:34.509402 18901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:34.513897 18901 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:44269
I20260620 14:53:34.513953 18968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:44269 every 8 connection(s)
I20260620 14:53:34.514353 18901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:34.515300 18969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:34.516955 18969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:34.523386 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 18901
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:34.523485 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:53:34.523684 5594 external_mini_cluster.cc:1428] Setting key 85a099673578aa3e698402328ccd5392
I20260620 14:53:34.527838 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.517797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.5.118.190:44269)
0620 14:53:34.517990 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.518000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.518119 (+ 119us) server_negotiation.cc:407] Connection header received
0620 14:53:34.518695 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.518697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.518717 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.518825 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.519387 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.520333 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.521240 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.521595 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.521727 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.524108 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.524115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.524115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.524122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.526055 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.526533 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.526537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.526537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.526627 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.526881 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.526882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.526882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.527133 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.527232 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.527419 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.527579 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260620 14:53:34.527841 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.517716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:44269 (local address 127.0.0.1:58686)
0620 14:53:34.517980 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.518025 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:34.518526 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:34.518830 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:34.518833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:34.519121 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:34.519282 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.519287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.520442 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.520443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.521135 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.521136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.521696 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.521697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.521766 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.522357 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:34.522366 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:34.523941 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:34.526176 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.526183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.526185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.526398 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.526701 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.526702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.526702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.526791 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.527210 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:34.527215 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:34.527280 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 14:53:34.527395 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:53:34.527644 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":92,"thread_start_us":99,"threads_started":1}
I20260620 14:53:34.529052 18969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:34.529446 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.529304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:35373 (local address 127.0.0.1:57156)
0620 14:53:34.529361 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.529422 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:35373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:53:34.529592 18920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.118.189:35373: connect: Connection refused (error 111)
I20260620 14:53:34.530025 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.529930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:35373 (local address 127.0.0.1:57164)
0620 14:53:34.529977 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.530002 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:35373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 14:53:34.530136 18969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.189:35373: Network error: Client connection negotiation failed: client connection to 127.5.118.189:35373: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.5.118.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab.
Entry for principal kudu/127.5.118.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab.
I20260620 14:53:34.551692 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.189:35373
--webserver_interface=127.5.118.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab
--principal=kudu/127.5.118.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:44269,127.5.118.189:35373,127.5.118.188:43341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:34.579389 18969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } attempt: 1
I20260620 14:53:34.579953 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.579782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:35373 (local address 127.0.0.1:57176)
0620 14:53:34.579857 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.579912 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:35373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:53:34.580415 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.580312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:35373 (local address 127.0.0.1:57182)
0620 14:53:34.580361 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.580393 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:35373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:53:34.580580 18969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.189:35373: Network error: Client connection negotiation failed: client connection to 127.5.118.189:35373: connect: Connection refused (error 111)
W20260620 14:53:34.634984 18976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:34.635201 18976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:34.635224 18976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:34.636497 18976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:34.636544 18976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:34.636559 18976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:34.636571 18976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:34.636583 18976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:34.638236 18976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:44269,127.5.118.189:35373,127.5.118.188:43341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab
--principal=kudu/127.5.118.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.189:35373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.118.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:34.638504 18976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:34.638751 18976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:34.643754 18981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:34.643754 18984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:34.643791 18982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:34.644248 18976 server_base.cc:1034] running on GCE node
I20260620 14:53:34.644505 18976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:34.644870 18976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:34.646044 18976 hybrid_clock.cc:648] HybridClock initialized: now 1781967214646015 us; error 44 us; skew 500 ppm
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:34.648732 18969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } attempt: 2
I20260620 14:53:34.649009 18976 init.cc:377] Logged in from keytab as kudu/127.5.118.189@KRBTEST.COM (short username kudu)
I20260620 14:53:34.649286 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.649096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:35373 (local address 127.0.0.1:57186)
0620 14:53:34.649203 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.649245 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:35373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 14:53:34.649645 18976 webserver.cc:460] Webserver started at http://127.5.118.189:44299/ using document root <none> and password file <none>
I20260620 14:53:34.649749 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.649592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:35373 (local address 127.0.0.1:57190)
0620 14:53:34.649680 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.649717 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:35373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:53:34.649889 18969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.189:35373: Network error: Client connection negotiation failed: client connection to 127.5.118.189:35373: connect: Connection refused (error 111)
I20260620 14:53:34.649931 18976 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:34.649988 18976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:34.650101 18976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:34.651163 18976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/data/instance:
uuid: "c4115f2e29f34b16be968c9b7a593ed6"
format_stamp: "Formatted at 2026-06-20 14:53:34 on dist-test-slave-2m5f"
server_key: "24136e0a69ebb0792a6f6967e75e03be"
server_key_iv: "38028598522a3d1cd511f82c102ec0cb"
server_key_version: "encryptionkey@0"
I20260620 14:53:34.651474 18976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/wal/instance:
uuid: "c4115f2e29f34b16be968c9b7a593ed6"
format_stamp: "Formatted at 2026-06-20 14:53:34 on dist-test-slave-2m5f"
server_key: "24136e0a69ebb0792a6f6967e75e03be"
server_key_iv: "38028598522a3d1cd511f82c102ec0cb"
server_key_version: "encryptionkey@0"
I20260620 14:53:34.652923 18976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:53:34.653775 18991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:34.653959 18976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:34.654053 18976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/wal
uuid: "c4115f2e29f34b16be968c9b7a593ed6"
format_stamp: "Formatted at 2026-06-20 14:53:34 on dist-test-slave-2m5f"
server_key: "24136e0a69ebb0792a6f6967e75e03be"
server_key_iv: "38028598522a3d1cd511f82c102ec0cb"
server_key_version: "encryptionkey@0"
I20260620 14:53:34.654134 18976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:34.669317 18976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:34.671772 18976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:34.671892 18976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:34.676031 18976 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.189:35373
I20260620 14:53:34.676105 19043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.189:35373 every 8 connection(s)
I20260620 14:53:34.676509 18976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
I20260620 14:53:34.676934 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 18976
I20260620 14:53:34.677066 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-1/wal/instance
I20260620 14:53:34.677277 5594 external_mini_cluster.cc:1428] Setting key 0e39442043c19a530045434dcd742994
I20260620 14:53:34.677709 19044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:34.679790 19044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.118.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.188@KRBTEST.COM" created.
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:34.693662 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.680668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.5.118.190:44269)
0620 14:53:34.680745 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.680748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.680998 (+ 250us) server_negotiation.cc:407] Connection header received
0620 14:53:34.681600 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.681602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.681615 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.681704 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.682328 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.683259 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.684332 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.684694 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.684784 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.689993 (+ 5209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.689995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.689995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.690004 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.691727 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.692332 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.692334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.692334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.692411 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.692750 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.692751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.692751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.692916 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.693001 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.693445 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.693543 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 14:53:34.693696 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.680653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:44269 (local address 127.0.0.1:58688)
0620 14:53:34.680837 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.680880 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:34.681453 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:34.681763 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:34.681766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:34.682027 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:34.682231 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.682236 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.683375 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.683376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.684245 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.684248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.684811 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.684812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.684935 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.685654 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:34.685663 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:34.689842 (+ 4179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:34.691842 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.691851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.691854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.692235 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.692534 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.692536 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.692536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.692663 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.693067 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:34.693073 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:34.693283 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 14:53:34.693368 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:53:34.693518 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260620 14:53:34.694408 19044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
I20260620 14:53:34.704411 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.694709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:35373 (local address 127.0.0.1:57198)
0620 14:53:34.694759 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.694771 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:34.694810 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:34.695116 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:34.695117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:34.695264 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:34.695309 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.695310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.696200 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.696201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.696701 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.696703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.697194 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.697195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.697267 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.697675 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:34.697679 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:34.700805 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:34.703109 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.703110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.703111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.703300 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.703579 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.703580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.703581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.703652 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.703985 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:34.703986 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:34.704031 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:53:34.704130 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:53:34.704278 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:53:34.704504 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.694782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.5.118.189:35373)
0620 14:53:34.694960 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.694963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.694980 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:53:34.695001 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.695003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.695017 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.695117 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.695382 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.696075 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.696795 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.697106 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.697195 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.700922 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.700944 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.700945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.700953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.702972 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.703410 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.703420 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.703421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.703500 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.703724 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.703724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.703725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.703898 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.704007 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.704166 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.704361 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
I20260620 14:53:34.705281 19044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:34.705708 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.705588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50082)
0620 14:53:34.705639 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.705686 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:53:34.705847 18995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
I20260620 14:53:34.706336 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.706234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50094)
0620 14:53:34.706280 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.706305 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:53:34.706432 19044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:43341: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab.
Entry for principal kudu/127.5.118.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab.
I20260620 14:53:34.707620 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.188:43341
--webserver_interface=127.5.118.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab
--principal=kudu/127.5.118.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:36239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:44269,127.5.118.189:35373,127.5.118.188:43341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:34.741072 18969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } attempt: 3
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
I20260620 14:53:34.749133 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.741494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:35373 (local address 127.0.0.1:57202)
0620 14:53:34.741591 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.741604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:34.741655 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:34.741817 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:34.741818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:34.741992 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:34.742059 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.742060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.742756 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.742757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.743333 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.743336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.743738 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.743739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.743824 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.744400 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:34.744405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:34.746105 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:34.747920 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.747921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.747922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.748137 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.748406 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.748407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.748408 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.748471 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.748776 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:34.748777 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:34.748833 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 14:53:34.748920 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:53:34.749031 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 14:53:34.749133 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.741597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.5.118.189:35373)
0620 14:53:34.741642 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.741644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.741656 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:53:34.741717 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.741718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.741730 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.741810 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.742152 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.742660 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.743438 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.743656 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.743740 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.746214 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.746216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.746216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.746224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.747791 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.748266 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.748268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.748268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.748332 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.748572 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.748573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.748574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.748700 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.748775 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.748944 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.749034 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:53:34.749533 18969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:34.749892 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.749779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50108)
0620 14:53:34.749830 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.749858 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:53:34.750187 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.750097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50124)
0620 14:53:34.750143 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.750166 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 14:53:34.750285 18969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:43341: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
I20260620 14:53:34.755652 19044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } attempt: 1
I20260620 14:53:34.756067 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.755942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50134)
0620 14:53:34.756002 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.756035 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:53:34.756458 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.756364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50142)
0620 14:53:34.756415 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.756440 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:53:34.756557 19044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:43341: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
I20260620 14:53:34.781450 18969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } attempt: 1
I20260620 14:53:34.781925 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.781788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50148)
0620 14:53:34.781852 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.781895 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:53:34.782354 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.782252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50160)
0620 14:53:34.782299 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.782326 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:53:34.782456 18969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:43341: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
W20260620 14:53:34.797713 19051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:34.797981 19051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:34.798020 19051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:34.799242 19051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:34.799316 19051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:34.799341 19051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:34.799362 19051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:34.799391 19051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:34.801098 19051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:44269,127.5.118.189:35373,127.5.118.188:43341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab
--principal=kudu/127.5.118.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.188:43341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.118.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:34.801398 19051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:34.801648 19051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:53:34.806908 19051 server_base.cc:1034] running on GCE node
W20260620 14:53:34.806874 19056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:34.806879 19059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:34.806850 19057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:34.807368 19051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:34.807718 19051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:34.808881 19051 hybrid_clock.cc:648] HybridClock initialized: now 1781967214808871 us; error 45 us; skew 500 ppm
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:34.811508 19051 init.cc:377] Logged in from keytab as kudu/127.5.118.188@KRBTEST.COM (short username kudu)
I20260620 14:53:34.812013 19051 webserver.cc:460] Webserver started at http://127.5.118.188:42703/ using document root <none> and password file <none>
I20260620 14:53:34.812309 19051 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:34.812358 19051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:34.812515 19051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:34.813716 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/data/instance:
uuid: "89acdc74198341d08b03896f85f7fdcb"
format_stamp: "Formatted at 2026-06-20 14:53:34 on dist-test-slave-2m5f"
server_key: "b72d5346b358a74cd19d387ad1a7a9d4"
server_key_iv: "9ed9cf7b8643b205648abdf343df09aa"
server_key_version: "encryptionkey@0"
I20260620 14:53:34.814031 19051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/wal/instance:
uuid: "89acdc74198341d08b03896f85f7fdcb"
format_stamp: "Formatted at 2026-06-20 14:53:34 on dist-test-slave-2m5f"
server_key: "b72d5346b358a74cd19d387ad1a7a9d4"
server_key_iv: "9ed9cf7b8643b205648abdf343df09aa"
server_key_version: "encryptionkey@0"
I20260620 14:53:34.815356 19051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:53:34.816107 19066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:34.816330 19051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:53:34.816419 19051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/wal
uuid: "89acdc74198341d08b03896f85f7fdcb"
format_stamp: "Formatted at 2026-06-20 14:53:34 on dist-test-slave-2m5f"
server_key: "b72d5346b358a74cd19d387ad1a7a9d4"
server_key_iv: "9ed9cf7b8643b205648abdf343df09aa"
server_key_version: "encryptionkey@0"
I20260620 14:53:34.816499 19051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:34.824697 19044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } attempt: 2
I20260620 14:53:34.825192 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.825014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50166)
0620 14:53:34.825078 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.825160 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:53:34.825582 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.825465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50182)
0620 14:53:34.825527 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.825554 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:53:34.825667 19044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:43341: Network error: Client connection negotiation failed: client connection to 127.5.118.188:43341: connect: Connection refused (error 111)
I20260620 14:53:34.834611 19051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:34.837001 19051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:34.837260 19051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:34.841390 19051 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.188:43341
I20260620 14:53:34.841423 19118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.188:43341 every 8 connection(s)
I20260620 14:53:34.841840 19051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
I20260620 14:53:34.842774 19119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:34.843149 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 19051
I20260620 14:53:34.843245 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967053866185-5594-0/minicluster-data/master-2/wal/instance
I20260620 14:53:34.843469 5594 external_mini_cluster.cc:1428] Setting key 9d07796c99728d66fbb71250fb8d83fe
I20260620 14:53:34.845185 19119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
I20260620 14:53:34.854403 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.846000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.5.118.190:44269)
0620 14:53:34.846058 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.846060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.846308 (+ 248us) server_negotiation.cc:407] Connection header received
0620 14:53:34.846379 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.846381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.846392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.846468 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.846970 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.847455 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.848380 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.848592 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.848681 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.850758 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.850760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.850760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.850768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.852630 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.853202 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.853204 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.853204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.853301 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.853566 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.853567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.853567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.853738 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.853815 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.854185 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.854292 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:53:34.854696 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.844085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.5.118.188:43341)
0620 14:53:34.844284 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.844295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.844357 (+ 62us) server_negotiation.cc:407] Connection header received
0620 14:53:34.844912 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.844922 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.844989 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.845130 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.846084 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.847024 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.847830 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.848307 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.848441 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.851189 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.851197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.851197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.851205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.853075 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.853633 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.853636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.853636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.853681 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.853906 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.853907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.853907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.854056 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.854195 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.854320 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.854381 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":107,"threads_started":1}
I20260620 14:53:34.854696 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.846018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:44269 (local address 127.0.0.1:58704)
0620 14:53:34.846213 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.846253 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:34.846325 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:34.846494 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:34.846497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:34.846769 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:34.846867 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.846872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.847544 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.847546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.848272 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.848276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.848692 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.848693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.848767 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.849186 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:34.849194 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:34.850659 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:34.852727 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.852733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.852736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.853103 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.853383 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.853384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.853384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.853491 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.853816 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:34.853820 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:34.853998 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 14:53:34.854123 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 14:53:34.854299 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":135,"threads_started":1}
I20260620 14:53:34.855302 19119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:34.857597 18969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } attempt: 2
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
I20260620 14:53:34.863737 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.855716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.5.118.190:44269)
0620 14:53:34.855756 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.855758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.855765 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:53:34.855784 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.855784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.855793 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.855858 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.856421 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.856862 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.857632 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.857903 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.857970 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.860518 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.860519 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.860520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.860526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.862291 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.862725 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.862726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.862726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.862787 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.863006 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.863007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.863008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.863148 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.863346 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.863559 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.863635 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:53:34.864404 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.855593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:35373 (local address 127.0.0.1:57204)
0620 14:53:34.855644 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.855654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:34.855695 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:34.855883 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:34.855884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:34.856042 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:34.856086 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.856088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.856946 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.856948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.857480 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.857483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.858159 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.858159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.858558 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.859730 (+ 1172us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:34.859734 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:34.861157 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:34.863010 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.863012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.863012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.863224 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.863455 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.863456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.863456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.863513 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.864032 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:34.864033 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:34.864081 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:53:34.864155 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:53:34.864274 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:53:34.864462 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.855724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.5.118.189:35373)
0620 14:53:34.855770 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.855771 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.855779 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:53:34.855803 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.855803 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.855812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.855865 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.856203 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.856848 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.857558 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.857850 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.857916 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.861294 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.861296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.861296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.861304 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.862915 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.863316 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.863317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.863318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.863378 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.863599 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.863600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.863601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.863739 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.863784 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.864262 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.864346 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:53:34.864809 19119 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:34.865109 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.857844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50190)
0620 14:53:34.857901 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.857911 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:34.857945 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:34.858133 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:34.858134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:34.858258 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:34.858288 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.858288 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.859101 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.859103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.859598 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.859599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.860103 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.860105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.860195 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.860611 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:34.860616 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:34.862200 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:34.863982 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.863983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.863984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.864243 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.864465 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.864466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.864466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.864524 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.864818 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:34.864819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:34.864858 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:53:34.864902 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 14:53:34.865003 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 14:53:34.865509 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.857980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.5.118.188:43341)
0620 14:53:34.858017 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.858019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.858026 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:53:34.858054 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.858055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.858064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.858123 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.858360 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.859018 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.859683 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.859986 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.860096 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.862295 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.862297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.862297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.862304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.863901 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.864331 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.864332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.864332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.864391 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.864604 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.864605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.864606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.864752 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.865302 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.865349 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.865425 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:53:34.867846 18969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457: Bootstrap starting.
I20260620 14:53:34.868744 18969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:34.869108 18969 log.cc:824] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
I20260620 14:53:34.870056 18969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457: No bootstrap required, opened a new log
I20260620 14:53:34.872439 18969 raft_consensus.cc:348] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } }
I20260620 14:53:34.872602 18969 raft_consensus.cc:374] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:34.872656 18969 raft_consensus.cc:729] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e63e9f5bbbe742e9a2764d84513f2457, State: Initialized, Role: FOLLOWER
I20260620 14:53:34.872810 18969 consensus_queue.cc:260] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } }
I20260620 14:53:34.873135 19128 sys_catalog.cc:455] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } } }
I20260620 14:53:34.873204 18969 sys_catalog.cc:564] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:34.873214 19128 sys_catalog.cc:458] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:34.873411 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.865121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.5.118.188:43341)
0620 14:53:34.865584 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.865586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.865594 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:53:34.865614 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.865615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.865625 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.865969 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.865998 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.866627 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.867626 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.867877 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.868152 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.870363 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.870365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.870365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.870372 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.871988 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.872394 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.872395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.872395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.872452 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.872664 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.872665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.872665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.872846 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.872919 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.873219 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.873310 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":99,"threads_started":1}
I20260620 14:53:34.873502 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.864499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57214 (local address 127.5.118.189:35373)
0620 14:53:34.864557 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.864559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.864713 (+ 154us) server_negotiation.cc:407] Connection header received
0620 14:53:34.864814 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.864815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.864826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.864854 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.865553 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.866198 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.867027 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.867238 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.867322 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.870422 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.870423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.870424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.870431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.872060 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.872464 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.872465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.872466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.872526 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.872719 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.872720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.872720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.872848 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.873041 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.873316 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.873392 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40,"thread_start_us":115,"threads_started":1}
I20260620 14:53:34.874089 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.865053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50198)
0620 14:53:34.865178 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.865188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:34.865226 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:34.865707 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:34.865708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:34.865843 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:34.865904 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.865905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.866721 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.866722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.867469 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.867472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.867960 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.867961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.868003 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.868469 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:34.868474 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:34.870266 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:34.872086 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.872088 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.872088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.872318 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.872512 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.872513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.872513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.872587 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.873026 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:34.873027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:34.873075 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:53:34.873864 (+ 789us) client_negotiation.cc:240] Negotiation successful
0620 14:53:34.873967 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
W20260620 14:53:34.875787 19139 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:53:34.875857 19139 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:53:34.875900 19119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb: Bootstrap starting.
I20260620 14:53:34.876616 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:53:34.876963 19119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:34.877264 19119 log.cc:824] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:34.878141 19119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb: No bootstrap required, opened a new log
I20260620 14:53:34.876632 18901 master_runner.cc:186] Error getting master registration for 127.5.118.189:35373: OK, instance_id { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" instance_seqno: 1781967214674838 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:53:34.879895 18901 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:53:34.880263 19119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } }
I20260620 14:53:34.880415 19119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:34.880465 19119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89acdc74198341d08b03896f85f7fdcb, State: Initialized, Role: FOLLOWER
I20260620 14:53:34.880584 19119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } }
I20260620 14:53:34.880897 19119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:34.880862 19149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } } }
I20260620 14:53:34.880949 19149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:34.886018 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.878247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58708 (local address 127.5.118.190:44269)
0620 14:53:34.878284 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.878286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.878492 (+ 206us) server_negotiation.cc:407] Connection header received
0620 14:53:34.878599 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.878600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.878612 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.878642 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.879235 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.879717 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.880678 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.880986 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.881056 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.882735 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.882737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.882738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.882745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.884601 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.884996 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.884998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.884998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.885059 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.885246 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.885247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.885247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.885372 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.885483 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.885851 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.885925 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:53:34.885911 19051 master_runner.cc:186] Error getting master registration for 127.5.118.189:35373: OK, instance_id { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" instance_seqno: 1781967214674838 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:53:34.889197 19051 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:53:34.889703 19160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:53:34.889803 19160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:53:34.892606 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.882261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.5.118.188:43341)
0620 14:53:34.882691 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.882693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.882702 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:53:34.882729 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.882730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.882739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.882795 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.883340 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.884398 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.886678 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.886957 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.887001 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.888657 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.888658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.888658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.888665 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.890762 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.891242 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.891243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.891244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.891297 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.891758 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.891759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.891760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.891889 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.892002 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.892424 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.892517 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408}
I20260620 14:53:34.894292 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.878105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.5.118.189:35373)
0620 14:53:34.878146 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.878148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.882534 (+ 4386us) server_negotiation.cc:407] Connection header received
0620 14:53:34.882614 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.882615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.882629 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.882682 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.884123 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.884834 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.889349 (+ 4515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.889675 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.889746 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.891244 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.891245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.891245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.891253 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.892848 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.893446 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.893447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.893448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.893522 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.893738 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.893739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.893739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.893875 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.893986 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.894122 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.894205 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:53:34.916814 19044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } attempt: 3
Jun 20 14:53:34 dist-test-slave-2m5f krb5kdc[18888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
I20260620 14:53:34.925290 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.917156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:43341 (local address 127.0.0.1:50224)
0620 14:53:34.917213 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:34.917225 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:34.917265 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:34.917421 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:34.917422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:34.917628 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:34.917684 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.917686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.918697 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.918699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.919551 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:34.919553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.920040 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.920041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:34.920104 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.920660 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:34.920666 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:34.922205 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:34.924120 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.924121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.924122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.924374 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.924602 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:34.924603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:34.924604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:34.924672 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:34.924950 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:34.924951 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:34.925002 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:53:34.925079 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 14:53:34.925183 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:53:34.925892 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:34.917235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.5.118.188:43341)
0620 14:53:34.917279 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:34.917281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:34.917291 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:53:34.917319 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:34.917320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:34.917331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:34.917401 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:34.917776 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.918577 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.919633 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:34.919961 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:34.920032 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:34.922300 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:34.922303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:34.922303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:34.922312 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:34.924003 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.924456 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.924457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.924457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.924527 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:34.924740 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:34.924741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:34.924741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:34.924878 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:34.924946 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:34.925204 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 14:53:34.925294 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:53:34.927783 19044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6: Bootstrap starting.
I20260620 14:53:34.928591 19044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:34.928896 19044 log.cc:824] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:34.929885 19044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6: No bootstrap required, opened a new log
I20260620 14:53:34.931975 19044 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } }
I20260620 14:53:34.932137 19044 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:34.932199 19044 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4115f2e29f34b16be968c9b7a593ed6, State: Initialized, Role: FOLLOWER
I20260620 14:53:34.932336 19044 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } }
I20260620 14:53:34.932678 19044 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:34.932641 19163 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } } }
I20260620 14:53:34.932857 19163 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:53:34.934597 19174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:53:34.934720 19174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:53:34.936779 18976 master_runner.cc:263] No leader master found from master c4115f2e29f34b16be968c9b7a593ed6
I20260620 14:53:34.936847 18976 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:53:35.000242 19128 raft_consensus.cc:482] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:35.000404 19128 raft_consensus.cc:504] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } }
I20260620 14:53:35.000736 19128 leader_election.cc:288] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c4115f2e29f34b16be968c9b7a593ed6 (127.5.118.189:35373), 89acdc74198341d08b03896f85f7fdcb (127.5.118.188:43341)
I20260620 14:53:35.001156 19093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e63e9f5bbbe742e9a2764d84513f2457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89acdc74198341d08b03896f85f7fdcb" is_pre_election: true
I20260620 14:53:35.001349 19093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e63e9f5bbbe742e9a2764d84513f2457 in term 0.
I20260620 14:53:35.001552 18920 leader_election.cc:302] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89acdc74198341d08b03896f85f7fdcb, e63e9f5bbbe742e9a2764d84513f2457; no voters:
I20260620 14:53:35.001920 19018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e63e9f5bbbe742e9a2764d84513f2457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4115f2e29f34b16be968c9b7a593ed6" is_pre_election: true
I20260620 14:53:35.002084 19018 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e63e9f5bbbe742e9a2764d84513f2457 in term 0.
I20260620 14:53:35.002334 19128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:35.002403 19128 raft_consensus.cc:482] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:35.002430 19128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:35.003422 19128 raft_consensus.cc:504] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } }
I20260620 14:53:35.003561 19128 leader_election.cc:288] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [CANDIDATE]: Term 1 election: Requested vote from peers c4115f2e29f34b16be968c9b7a593ed6 (127.5.118.189:35373), 89acdc74198341d08b03896f85f7fdcb (127.5.118.188:43341)
I20260620 14:53:35.003981 19018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e63e9f5bbbe742e9a2764d84513f2457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4115f2e29f34b16be968c9b7a593ed6"
I20260620 14:53:35.004066 19018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:35.005020 19018 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e63e9f5bbbe742e9a2764d84513f2457 in term 1.
I20260620 14:53:35.005141 19093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e63e9f5bbbe742e9a2764d84513f2457" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89acdc74198341d08b03896f85f7fdcb"
I20260620 14:53:35.005229 19093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:35.006067 19093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e63e9f5bbbe742e9a2764d84513f2457 in term 1.
I20260620 14:53:35.006260 18920 leader_election.cc:302] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89acdc74198341d08b03896f85f7fdcb, e63e9f5bbbe742e9a2764d84513f2457; no voters:
I20260620 14:53:35.007359 19128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:35.007521 19128 raft_consensus.cc:686] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [term 1 LEADER]: Becoming Leader. State: Replica: e63e9f5bbbe742e9a2764d84513f2457, State: Running, Role: LEADER
I20260620 14:53:35.007622 19128 consensus_queue.cc:237] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } }
I20260620 14:53:35.008041 19128 sys_catalog.cc:455] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e63e9f5bbbe742e9a2764d84513f2457. Latest consensus state: current_term: 1 leader_uuid: "e63e9f5bbbe742e9a2764d84513f2457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } } }
I20260620 14:53:35.008139 19128 sys_catalog.cc:458] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:35.008867 19178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:35.009091 19178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 14:53:35.010222 18922 tablet.cc:2369] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:35.010788 19093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [term 1 FOLLOWER]: Refusing update from remote peer e63e9f5bbbe742e9a2764d84513f2457: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:35.011065 19018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [term 1 FOLLOWER]: Refusing update from remote peer e63e9f5bbbe742e9a2764d84513f2457: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:35.011374 19128 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:35.012789 19163 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e63e9f5bbbe742e9a2764d84513f2457. Latest consensus state: current_term: 1 leader_uuid: "e63e9f5bbbe742e9a2764d84513f2457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } } }
I20260620 14:53:35.012890 19163 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:35.012972 19128 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:35.014189 19149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e63e9f5bbbe742e9a2764d84513f2457. Latest consensus state: current_term: 1 leader_uuid: "e63e9f5bbbe742e9a2764d84513f2457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } } }
I20260620 14:53:35.014293 19149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:35.016472 19128 sys_catalog.cc:455] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e63e9f5bbbe742e9a2764d84513f2457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } } }
I20260620 14:53:35.016566 19128 sys_catalog.cc:458] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:35.016995 19149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e63e9f5bbbe742e9a2764d84513f2457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } } }
I20260620 14:53:35.017086 19149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:35.017298 19179 mvcc.cc:205] Tried to move back new op lower bound from 7298937712681238528 to 7298937712671903744. Current Snapshot: MvccSnapshot[applied={T|T < 7298937712681238528}]
I20260620 14:53:35.017783 19128 sys_catalog.cc:455] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e63e9f5bbbe742e9a2764d84513f2457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } } }
I20260620 14:53:35.017866 19128 sys_catalog.cc:458] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:35.018287 19163 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e63e9f5bbbe742e9a2764d84513f2457" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e63e9f5bbbe742e9a2764d84513f2457" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44269 } } peers { permanent_uuid: "c4115f2e29f34b16be968c9b7a593ed6" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 35373 } } peers { permanent_uuid: "89acdc74198341d08b03896f85f7fdcb" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 43341 } } }
I20260620 14:53:35.018385 19163 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:35.018705 19182 mvcc.cc:205] Tried to move back new op lower bound from 7298937712681238528 to 7298937712671903744. Current Snapshot: MvccSnapshot[applied={T|T < 7298937712681238528}]
I20260620 14:53:35.018844 19180 mvcc.cc:205] Tried to move back new op lower bound from 7298937712681238528 to 7298937712671903744. Current Snapshot: MvccSnapshot[applied={T|T < 7298937712681238528}]
I20260620 14:53:35.020779 19178 catalog_manager.cc:1347] Generated new cluster ID: 2d259dc10cf348d19f3669113e7c96f7
I20260620 14:53:35.020840 19178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:35.045714 19178 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:35.046597 19178 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:35.053856 19178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e63e9f5bbbe742e9a2764d84513f2457: Generated new TSK 0
I20260620 14:53:35.054069 19178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:53:35.890357 19160 catalog_manager.cc:1259] Loaded cluster ID: 2d259dc10cf348d19f3669113e7c96f7
I20260620 14:53:35.890431 19160 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb: loading cluster ID for follower catalog manager: success
I20260620 14:53:35.891327 19160 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb: acquiring CA information for follower catalog manager: success
I20260620 14:53:35.891618 19160 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 89acdc74198341d08b03896f85f7fdcb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:53:35.936079 19174 catalog_manager.cc:1259] Loaded cluster ID: 2d259dc10cf348d19f3669113e7c96f7
I20260620 14:53:35.936159 19174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6: loading cluster ID for follower catalog manager: success
I20260620 14:53:35.937033 19174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6: acquiring CA information for follower catalog manager: success
I20260620 14:53:35.937338 19174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4115f2e29f34b16be968c9b7a593ed6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:53:36.110663 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 18901
I20260620 14:53:36.116566 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 18976
I20260620 14:53:36.121954 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 19051
2026-06-20T14:53:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2938 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:36 dist-test-slave-2m5f krb5kdc[19194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:36 dist-test-slave-2m5f krb5kdc[19194](info): set up 2 sockets
Jun 20 14:53:36 dist-test-slave-2m5f krb5kdc[19194](info): commencing operation
krb5kdc: starting...
W20260620 14:53:37.184756 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:53:37 dist-test-slave-2m5f krb5kdc[19194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:53:37.311338 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:33159
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:33159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:37.398036 19207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:37.398298 19207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:37.398339 19207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:37.399639 19207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:37.399720 19207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:37.399742 19207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:37.399761 19207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:37.399791 19207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:37.401377 19207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:33159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:33159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:37.401687 19207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:37.401933 19207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:37.407172 19212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:37.407172 19215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:37.407238 19213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:37.407244 19207 server_base.cc:1034] running on GCE node
I20260620 14:53:37.407784 19207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:37.408120 19207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:37.409325 19207 hybrid_clock.cc:648] HybridClock initialized: now 1781967217409303 us; error 42 us; skew 500 ppm
Jun 20 14:53:37 dist-test-slave-2m5f krb5kdc[19194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967217, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:37.411942 19207 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:53:37.412492 19207 webserver.cc:460] Webserver started at http://127.5.118.190:44851/ using document root <none> and password file <none>
I20260620 14:53:37.412725 19207 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:37.412793 19207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:37.412951 19207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:37.414160 19207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "e10412484a69462c8ad5473df9e5eeef"
format_stamp: "Formatted at 2026-06-20 14:53:37 on dist-test-slave-2m5f"
server_key: "4418c051779f7803b9d8831b669d0f30"
server_key_iv: "edea4fee49c3796ae496c696e4d71a92"
server_key_version: "encryptionkey@0"
I20260620 14:53:37.414546 19207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "e10412484a69462c8ad5473df9e5eeef"
format_stamp: "Formatted at 2026-06-20 14:53:37 on dist-test-slave-2m5f"
server_key: "4418c051779f7803b9d8831b669d0f30"
server_key_iv: "edea4fee49c3796ae496c696e4d71a92"
server_key_version: "encryptionkey@0"
I20260620 14:53:37.415922 19207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:53:37.416795 19222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:37.416985 19207 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:53:37.417101 19207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "e10412484a69462c8ad5473df9e5eeef"
format_stamp: "Formatted at 2026-06-20 14:53:37 on dist-test-slave-2m5f"
server_key: "4418c051779f7803b9d8831b669d0f30"
server_key_iv: "edea4fee49c3796ae496c696e4d71a92"
server_key_version: "encryptionkey@0"
I20260620 14:53:37.417179 19207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:37.426077 19207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:37.428347 19207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:37.428516 19207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:37.432646 19207 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:33159
I20260620 14:53:37.432678 19274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:33159 every 8 connection(s)
I20260620 14:53:37.433063 19207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:37.433988 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:37.436373 19275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef: Bootstrap starting.
I20260620 14:53:37.436940 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 19207
I20260620 14:53:37.437057 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:53:37.437100 19275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:37.437290 5594 external_mini_cluster.cc:1428] Setting key 6e32ea7b5db5522993f2a9314cb7251a
I20260620 14:53:37.437448 19275 log.cc:824] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:37.438328 19275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef: No bootstrap required, opened a new log
I20260620 14:53:37.440809 19275 raft_consensus.cc:348] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10412484a69462c8ad5473df9e5eeef" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 33159 } }
I20260620 14:53:37.441004 19275 raft_consensus.cc:374] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:37.441045 19275 raft_consensus.cc:729] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e10412484a69462c8ad5473df9e5eeef, State: Initialized, Role: FOLLOWER
I20260620 14:53:37.441169 19275 consensus_queue.cc:260] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10412484a69462c8ad5473df9e5eeef" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 33159 } }
I20260620 14:53:37.441246 19275 raft_consensus.cc:388] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:37.441274 19275 raft_consensus.cc:482] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:37.441318 19275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:37.442010 19275 raft_consensus.cc:504] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10412484a69462c8ad5473df9e5eeef" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 33159 } }
I20260620 14:53:37.442102 19275 leader_election.cc:302] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e10412484a69462c8ad5473df9e5eeef; no voters:
I20260620 14:53:37.442346 19275 leader_election.cc:288] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:37.442373 19280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:37.442646 19280 raft_consensus.cc:686] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [term 1 LEADER]: Becoming Leader. State: Replica: e10412484a69462c8ad5473df9e5eeef, State: Running, Role: LEADER
I20260620 14:53:37.442701 19275 sys_catalog.cc:564] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:37.443202 19280 consensus_queue.cc:237] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10412484a69462c8ad5473df9e5eeef" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 33159 } }
I20260620 14:53:37.443619 19280 sys_catalog.cc:455] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [sys.catalog]: SysCatalogTable state changed. Reason: New leader e10412484a69462c8ad5473df9e5eeef. Latest consensus state: current_term: 1 leader_uuid: "e10412484a69462c8ad5473df9e5eeef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10412484a69462c8ad5473df9e5eeef" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 33159 } } }
I20260620 14:53:37.444025 19280 sys_catalog.cc:458] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:37.444386 19289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:37.444890 19289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:53:37 dist-test-slave-2m5f krb5kdc[19194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967217, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:37.443943 19281 sys_catalog.cc:455] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e10412484a69462c8ad5473df9e5eeef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e10412484a69462c8ad5473df9e5eeef" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 33159 } } }
I20260620 14:53:37.445540 19281 sys_catalog.cc:458] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:37.446768 19289 catalog_manager.cc:1347] Generated new cluster ID: 6ff38a5877ec42009bf94f59287d2d75
I20260620 14:53:37.446825 19289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:37.449323 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:37.437829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.5.118.190:33159)
0620 14:53:37.438006 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:37.438016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:37.438051 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:53:37.438556 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:37.438563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:37.438614 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:37.438734 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:37.439547 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.440592 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.441673 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:37.442285 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:37.442400 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:37.445732 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:37.445737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:37.445737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:37.445750 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:37.447674 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:37.448052 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:37.448055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:37.448055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:37.448151 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:37.448439 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:37.448440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:37.448441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:37.448682 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:37.448856 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:37.448971 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:53:37.449125 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260620 14:53:37.453294 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:53:37.466547 19289 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:37.467235 19289 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:37.473851 19289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e10412484a69462c8ad5473df9e5eeef: Generated new TSK 0
I20260620 14:53:37.474084 19289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:53:37.507364 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 19207
2026-06-20T14:53:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1383 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:37 dist-test-slave-2m5f krb5kdc[19302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:37 dist-test-slave-2m5f krb5kdc[19302](info): set up 2 sockets
Jun 20 14:53:37 dist-test-slave-2m5f krb5kdc[19302](info): commencing operation
krb5kdc: starting...
W20260620 14:53:38.560361 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:53:38 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:53:38.683362 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:38567
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:38567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:38.766193 19315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:38.766455 19315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:38.766501 19315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:38.767838 19315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:38.768016 19315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:38.768050 19315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:38.768090 19315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:38.768116 19315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:38.769843 19315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:38567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:38567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:38.770169 19315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:38.770432 19315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:53:38.776081 19315 server_base.cc:1034] running on GCE node
W20260620 14:53:38.775950 19323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:38.776198 19321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:38.775950 19320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:38.776607 19315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:38.776970 19315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:38.778146 19315 hybrid_clock.cc:648] HybridClock initialized: now 1781967218778133 us; error 46 us; skew 500 ppm
Jun 20 14:53:38 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967218, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:38.780862 19315 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:53:38.781404 19315 webserver.cc:460] Webserver started at http://127.5.118.190:37045/ using document root <none> and password file <none>
I20260620 14:53:38.781664 19315 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:38.781744 19315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:38.781908 19315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:38.783104 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "8ab3250e20914d13bcd6b12f55b77d13"
format_stamp: "Formatted at 2026-06-20 14:53:38 on dist-test-slave-2m5f"
server_key: "27363d0b8d43069838dc07c33b95d992"
server_key_iv: "934e5f6a0b6792961c771eac971b435a"
server_key_version: "encryptionkey@0"
I20260620 14:53:38.783509 19315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "8ab3250e20914d13bcd6b12f55b77d13"
format_stamp: "Formatted at 2026-06-20 14:53:38 on dist-test-slave-2m5f"
server_key: "27363d0b8d43069838dc07c33b95d992"
server_key_iv: "934e5f6a0b6792961c771eac971b435a"
server_key_version: "encryptionkey@0"
I20260620 14:53:38.785152 19315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:53:38.785982 19330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:38.786217 19315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:38.786335 19315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "8ab3250e20914d13bcd6b12f55b77d13"
format_stamp: "Formatted at 2026-06-20 14:53:38 on dist-test-slave-2m5f"
server_key: "27363d0b8d43069838dc07c33b95d992"
server_key_iv: "934e5f6a0b6792961c771eac971b435a"
server_key_version: "encryptionkey@0"
I20260620 14:53:38.786419 19315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:38.804883 19315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:38.807013 19315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:38.807214 19315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:38.811667 19315 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:38567
I20260620 14:53:38.811977 19382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:38567 every 8 connection(s)
I20260620 14:53:38.812237 19315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:38.813215 19383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:38.815605 19383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Bootstrap starting.
I20260620 14:53:38.816293 19383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:38.816583 19383 log.cc:824] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:38.817310 19383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: No bootstrap required, opened a new log
I20260620 14:53:38.818534 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 19315
I20260620 14:53:38.818723 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:53:38.819010 5594 external_mini_cluster.cc:1428] Setting key 0d1c1721a7692cb212f62de911bff3b8
I20260620 14:53:38.819043 19383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:38.819268 19383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:38.819314 19383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab3250e20914d13bcd6b12f55b77d13, State: Initialized, Role: FOLLOWER
I20260620 14:53:38.819478 19383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:38.819583 19383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:38.819621 19383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:38.819729 19383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:38.821122 19383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:38.821234 19383 leader_election.cc:302] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ab3250e20914d13bcd6b12f55b77d13; no voters:
I20260620 14:53:38.821419 19383 leader_election.cc:288] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:38.821449 19388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:38.821566 19388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 1 LEADER]: Becoming Leader. State: Replica: 8ab3250e20914d13bcd6b12f55b77d13, State: Running, Role: LEADER
I20260620 14:53:38.821699 19383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:38.821717 19388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:38.822151 19389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ab3250e20914d13bcd6b12f55b77d13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } } }
I20260620 14:53:38.822257 19389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:38.822419 19390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ab3250e20914d13bcd6b12f55b77d13. Latest consensus state: current_term: 1 leader_uuid: "8ab3250e20914d13bcd6b12f55b77d13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } } }
I20260620 14:53:38.822495 19390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:38.822767 19395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:38.823361 19395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:38.825376 19395 catalog_manager.cc:1347] Generated new cluster ID: 67e63a1ee40d447e90ed4fc9327f7fec
I20260620 14:53:38.825440 19395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:53:38 dist-test-slave-2m5f krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:38.832288 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:38.819606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.5.118.190:38567)
0620 14:53:38.819846 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:38.819864 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:38.819904 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:53:38.820026 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:38.820032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:38.820091 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:38.820303 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:38.821135 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.822591 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.823852 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:38.824531 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:38.824667 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:38.827831 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:38.827835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:38.827836 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:38.827849 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:38.830055 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:38.830494 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:38.830496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:38.830497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:38.830634 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:38.830886 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:38.830887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:38.830887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:38.831255 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:38.831415 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:38.831892 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 14:53:38.832040 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":111,"threads_started":1}
I20260620 14:53:38.858907 19395 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:38.859552 19395 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:38.871755 19395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Generated new TSK 0
I20260620 14:53:38.871996 19395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:53:38.907804 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38567
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:38.996364 19409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:38.996582 19409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:38.996610 19409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:38.997766 19409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:38.997807 19409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:38.997833 19409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:53:38.999265 19409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:38.999461 19409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:38.999645 19409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:39.004681 19415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:39.004731 19417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:39.004837 19414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:39.004838 19409 server_base.cc:1034] running on GCE node
I20260620 14:53:39.005159 19409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:39.005379 19409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:39.006582 19409 hybrid_clock.cc:648] HybridClock initialized: now 1781967219006565 us; error 26 us; skew 500 ppm
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:39.009414 19409 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:53:39.009887 19409 webserver.cc:460] Webserver started at http://127.5.118.129:34235/ using document root <none> and password file <none>
I20260620 14:53:39.010133 19409 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:39.010205 19409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:39.010354 19409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:39.011431 19409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "61b7957b2eff4649b72bf7295887eeb2"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "57e0fe9d0b4f270509aedead5b00c12c"
server_key_iv: "5bbbbe5443e90519b679a0477e5f7a4e"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.011798 19409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "61b7957b2eff4649b72bf7295887eeb2"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "57e0fe9d0b4f270509aedead5b00c12c"
server_key_iv: "5bbbbe5443e90519b679a0477e5f7a4e"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.013437 19409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:53:39.014195 19424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.014389 19409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.014494 19409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "61b7957b2eff4649b72bf7295887eeb2"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "57e0fe9d0b4f270509aedead5b00c12c"
server_key_iv: "5bbbbe5443e90519b679a0477e5f7a4e"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.014580 19409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:39.032912 19409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:39.035097 19409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:39.035279 19409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:39.035537 19409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:39.035859 19409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:39.035917 19409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.035969 19409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:39.036017 19409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.042021 19409 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:38877
I20260620 14:53:39.042074 19537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:38877 every 8 connection(s)
I20260620 14:53:39.042407 19409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:39.043277 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 19409
I20260620 14:53:39.043402 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:53:39.043637 5594 external_mini_cluster.cc:1428] Setting key 7dcad4b721650d2f2384f487712aeb06
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
I20260620 14:53:39.056803 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.043184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:54269 (local address 127.5.118.190:38567)
0620 14:53:39.043242 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.043244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.043485 (+ 241us) server_negotiation.cc:407] Connection header received
0620 14:53:39.044250 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.044251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.044263 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.044315 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:39.045730 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.048378 (+ 2648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.050103 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.050445 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.050531 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.053556 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:39.053558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:39.053558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:39.053567 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:39.055331 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.055752 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.055752 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.055753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.055806 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.056054 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.056056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.056056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.056235 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:39.056302 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.056579 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.056672 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:53:39.057068 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.043068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:54269)
0620 14:53:39.043288 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.043386 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.044108 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.044400 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.044407 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.045294 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:39.045598 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.045605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.048480 (+ 2875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.048481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.049994 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.049998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.050564 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.050565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.050737 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.051390 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:39.051404 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:39.053420 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:39.055415 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:39.055423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:39.055425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:39.055682 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:39.055849 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:39.055849 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:39.055850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:39.055938 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:39.056309 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:39.056314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:39.056447 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.056573 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.056770 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":113,"threads_started":1}
I20260620 14:53:39.057669 19538 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38567
I20260620 14:53:39.057782 19538 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:39.057983 19538 heartbeater.cc:511] Master 127.5.118.190:38567 requested a full tablet report, sending...
I20260620 14:53:39.058674 19347 ts_manager.cc:194] Registered new tserver with Master: 61b7957b2eff4649b72bf7295887eeb2 (127.5.118.129:38877)
I20260620 14:53:39.059290 19347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:54269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:53:39.073109 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38567
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:39.157861 19543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:39.158092 19543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:39.158118 19543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:39.159380 19543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:39.159442 19543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:39.159472 19543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:53:39.161159 19543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:39.161494 19543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:39.161772 19543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:39.167207 19551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:39.167341 19543 server_base.cc:1034] running on GCE node
W20260620 14:53:39.167244 19549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:39.167444 19548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:39.167682 19543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:39.167945 19543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:39.169111 19543 hybrid_clock.cc:648] HybridClock initialized: now 1781967219169097 us; error 27 us; skew 500 ppm
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:39.171785 19543 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:53:39.172365 19543 webserver.cc:460] Webserver started at http://127.5.118.130:45019/ using document root <none> and password file <none>
I20260620 14:53:39.172585 19543 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:39.172631 19543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:39.172722 19543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:39.173761 19543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "ea490d5a70a94ba89549a37c83fff72a"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "f345e10e929e568e4d3d4e3667710fc0"
server_key_iv: "e8d8409166edd4837cf22a82198e85b9"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.174062 19543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "ea490d5a70a94ba89549a37c83fff72a"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "f345e10e929e568e4d3d4e3667710fc0"
server_key_iv: "e8d8409166edd4837cf22a82198e85b9"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.175370 19543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:53:39.176095 19558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.176312 19543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:53:39.176411 19543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "ea490d5a70a94ba89549a37c83fff72a"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "f345e10e929e568e4d3d4e3667710fc0"
server_key_iv: "e8d8409166edd4837cf22a82198e85b9"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.176481 19543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:39.184522 19543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:39.186942 19543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:39.187076 19543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:39.187259 19543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:39.187606 19543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:39.187644 19543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.187667 19543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:39.187716 19543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.193738 19543 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:33551
I20260620 14:53:39.193873 19671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:33551 every 8 connection(s)
I20260620 14:53:39.194176 19543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:39.198534 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 19543
I20260620 14:53:39.198663 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:53:39.198868 5594 external_mini_cluster.cc:1428] Setting key d96fcb24b8b47ca46717641c4d5b25ea
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:39.205099 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.195136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:35823 (local address 127.5.118.190:38567)
0620 14:53:39.195186 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.195188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.195335 (+ 147us) server_negotiation.cc:407] Connection header received
0620 14:53:39.195859 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.195860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.195870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.195924 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:39.197051 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.197566 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.198650 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.199039 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.199100 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.201934 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:39.201936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:39.201936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:39.201942 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:39.203412 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.203996 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.203997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.203998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.204051 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.204379 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.204380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.204380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.204508 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:39.204564 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.204947 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.205031 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:53:39.205428 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.194956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:35823)
0620 14:53:39.195223 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.195264 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.195760 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.196005 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.196008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.196723 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:39.196948 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.196953 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.197666 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.197666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.198534 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.198536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.199144 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.199145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.199299 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.199950 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:39.199962 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:39.201825 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:39.203524 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:39.203541 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:39.203544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:39.203907 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:39.204129 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:39.204130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:39.204130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:39.204305 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:39.204583 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:39.204591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:39.204792 (+ 201us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.204919 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.205126 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":75,"threads_started":1}
I20260620 14:53:39.206074 19672 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38567
I20260620 14:53:39.206168 19672 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:39.206380 19672 heartbeater.cc:511] Master 127.5.118.190:38567 requested a full tablet report, sending...
I20260620 14:53:39.206868 19347 ts_manager.cc:194] Registered new tserver with Master: ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551)
I20260620 14:53:39.207392 19347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:35823
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:53:39.227026 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38567
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:39.310204 19677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:39.310451 19677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:39.310477 19677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:39.311791 19677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:39.311839 19677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:39.311868 19677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:53:39.313443 19677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:39.313647 19677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:39.313835 19677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:39.319136 19682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:39.319151 19683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:39.319298 19685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:39.319566 19677 server_base.cc:1034] running on GCE node
I20260620 14:53:39.319849 19677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:39.320068 19677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:39.321221 19677 hybrid_clock.cc:648] HybridClock initialized: now 1781967219321218 us; error 55 us; skew 500 ppm
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:39.324771 19677 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:53:39.325415 19677 webserver.cc:460] Webserver started at http://127.5.118.131:34963/ using document root <none> and password file <none>
I20260620 14:53:39.325743 19677 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:39.325820 19677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:39.325971 19677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:39.327059 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "4106f645342b444fb6ed90c80b02bb18"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "8441e4645b34b62fb9af2419e5706158"
server_key_iv: "983e0ed7073a2423f29208b1633845e7"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.327371 19677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "4106f645342b444fb6ed90c80b02bb18"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "8441e4645b34b62fb9af2419e5706158"
server_key_iv: "983e0ed7073a2423f29208b1633845e7"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.328838 19677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:53:39.329695 19692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.329929 19677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:39.330024 19677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "4106f645342b444fb6ed90c80b02bb18"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "8441e4645b34b62fb9af2419e5706158"
server_key_iv: "983e0ed7073a2423f29208b1633845e7"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.330152 19677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:39.350834 19677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:39.353367 19677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:39.353487 19677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:39.353693 19677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:39.354045 19677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:39.354085 19677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.354110 19677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:39.354121 19677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.360529 19677 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:40805
I20260620 14:53:39.360621 19805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:40805 every 8 connection(s)
I20260620 14:53:39.360985 19677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:39.362293 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 19677
I20260620 14:53:39.362368 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:53:39.362547 5594 external_mini_cluster.cc:1428] Setting key ae6bce4e711e9c0593850e33cf5a4b72
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:39.371413 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.361828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:56983 (local address 127.5.118.190:38567)
0620 14:53:39.361914 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.361916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.362263 (+ 347us) server_negotiation.cc:407] Connection header received
0620 14:53:39.362961 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.362962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.362975 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.363096 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:39.364345 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.364840 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.365910 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.366134 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.366200 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.368616 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:39.368617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:39.368618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:39.368625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:39.370005 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.370434 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.370435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.370436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.370490 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.370801 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.370802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.370802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.370930 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:39.370992 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.371271 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.371343 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:53:39.371731 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.361806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:56983)
0620 14:53:39.362105 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.362159 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.362807 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.363224 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.363230 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.363981 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:39.364246 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.364253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.364933 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.364933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.365802 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.365806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.366224 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.366225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.366340 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.366942 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:39.366965 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:39.368518 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:39.370110 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:39.370116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:39.370118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:39.370352 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:39.370583 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:39.370584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:39.370584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:39.370706 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:39.371003 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:39.371009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:39.371141 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.371236 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.371415 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260620 14:53:39.372403 19806 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38567
I20260620 14:53:39.372519 19806 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:39.372766 19806 heartbeater.cc:511] Master 127.5.118.190:38567 requested a full tablet report, sending...
I20260620 14:53:39.373198 19347 ts_manager.cc:194] Registered new tserver with Master: 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805)
I20260620 14:53:39.373556 19347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:56983
I20260620 14:53:39.374428 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:53:39.381906 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.375800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.5.118.190:38567)
0620 14:53:39.375856 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.375858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.375921 (+ 63us) server_negotiation.cc:407] Connection header received
0620 14:53:39.375978 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.375979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.375986 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.376039 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:39.376628 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.377152 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.377854 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.378120 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.378176 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.379294 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:39.379296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:39.379296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:39.379302 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:39.380833 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.381207 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.381208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.381208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.381258 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.381453 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.381454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.381454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.381598 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:39.381678 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.381779 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.381840 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:53:39.383695 19347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:53:39.385484 19347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:39.392571 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.387952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:33551 (local address 127.0.0.1:38394)
0620 14:53:39.388895 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.388906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.388994 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.389176 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.389177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.389179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:39.389227 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.389229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.390237 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.390238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.391333 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.391336 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.392266 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.392268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.392340 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.392353 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.392417 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.392461 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":149,"threads_started":1}
I20260620 14:53:39.392598 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.388234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38394 (local address 127.5.118.130:33551)
0620 14:53:39.388425 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.388430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.388964 (+ 534us) server_negotiation.cc:407] Connection header received
0620 14:53:39.389068 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.389070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.389086 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.389181 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:39.389290 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.390136 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.391428 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.392152 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.392266 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.392331 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.392455 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.392511 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":80,"threads_started":1}
I20260620 14:53:39.392918 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.387942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:40805 (local address 127.0.0.1:39102)
0620 14:53:39.388167 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.388232 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.388304 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.388792 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.388796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.388799 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:39.388915 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.388921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.390348 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.390350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.391577 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.391580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.392539 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.392540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.392598 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.392602 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.392810 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.392854 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":54,"thread_start_us":153,"threads_started":1}
I20260620 14:53:39.393177 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.388235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.5.118.131:40805)
0620 14:53:39.388615 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.388620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.388633 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:53:39.388671 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.388673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.388691 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.388786 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:39.388985 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.390219 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.391671 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.392454 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.392523 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.392584 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.393052 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.393097 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":76,"threads_started":1}
I20260620 14:53:39.393410 19606 tablet_service.cc:1467] Processing CreateTablet for tablet ac37378591b341078262282705ac40de (DEFAULT_TABLE table=test-table [id=14fcf0911d7b473297c3dc32eea26ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:39.393787 19606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac37378591b341078262282705ac40de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:39.393805 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.387942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:38877 (local address 127.0.0.1:52584)
0620 14:53:39.388192 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.388232 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.388304 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.388874 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.388875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.388877 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:39.388934 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.388937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.390370 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.390371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.391577 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.391580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.393574 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.393575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.393639 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.393643 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.393701 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.393739 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":44,"thread_start_us":53,"threads_started":1}
I20260620 14:53:39.393831 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.388418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52584 (local address 127.5.118.129:38877)
0620 14:53:39.388608 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.388612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.388623 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:53:39.388661 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.388665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.388701 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.388777 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:39.389028 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.390235 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.392702 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.393471 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.393570 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.393625 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.393716 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.393753 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260620 14:53:39.396629 19823 tablet_bootstrap.cc:492] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a: Bootstrap starting.
I20260620 14:53:39.397308 19823 tablet_bootstrap.cc:654] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:39.397208 19740 tablet_service.cc:1467] Processing CreateTablet for tablet ac37378591b341078262282705ac40de (DEFAULT_TABLE table=test-table [id=14fcf0911d7b473297c3dc32eea26ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:39.397330 19472 tablet_service.cc:1467] Processing CreateTablet for tablet ac37378591b341078262282705ac40de (DEFAULT_TABLE table=test-table [id=14fcf0911d7b473297c3dc32eea26ea6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:39.397568 19740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac37378591b341078262282705ac40de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:39.397715 19472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac37378591b341078262282705ac40de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:39.397953 19823 log.cc:824] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:39.398752 19823 tablet_bootstrap.cc:492] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a: No bootstrap required, opened a new log
I20260620 14:53:39.398826 19823 ts_tablet_manager.cc:1397] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:39.400633 19825 tablet_bootstrap.cc:492] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Bootstrap starting.
I20260620 14:53:39.400961 19823 raft_consensus.cc:348] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.401096 19826 tablet_bootstrap.cc:492] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18: Bootstrap starting.
I20260620 14:53:39.401180 19823 raft_consensus.cc:374] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:39.401230 19823 raft_consensus.cc:729] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea490d5a70a94ba89549a37c83fff72a, State: Initialized, Role: FOLLOWER
I20260620 14:53:39.401511 19825 tablet_bootstrap.cc:654] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:39.401495 19823 consensus_queue.cc:260] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.401737 19826 tablet_bootstrap.cc:654] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:39.401804 19823 ts_tablet_manager.cc:1428] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:39.401933 19672 heartbeater.cc:503] Master 127.5.118.190:38567 was elected leader, sending a full tablet report...
I20260620 14:53:39.402114 19825 log.cc:824] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:39.402369 19826 log.cc:824] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:39.402930 19825 tablet_bootstrap.cc:492] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: No bootstrap required, opened a new log
I20260620 14:53:39.402999 19825 ts_tablet_manager.cc:1397] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:39.404524 19825 raft_consensus.cc:348] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.404714 19825 raft_consensus.cc:374] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:39.404754 19825 raft_consensus.cc:729] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b7957b2eff4649b72bf7295887eeb2, State: Initialized, Role: FOLLOWER
I20260620 14:53:39.404867 19826 tablet_bootstrap.cc:492] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18: No bootstrap required, opened a new log
I20260620 14:53:39.404875 19825 consensus_queue.cc:260] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.404937 19826 ts_tablet_manager.cc:1397] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:53:39.405155 19825 ts_tablet_manager.cc:1428] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:39.405292 19538 heartbeater.cc:503] Master 127.5.118.190:38567 was elected leader, sending a full tablet report...
I20260620 14:53:39.406996 19826 raft_consensus.cc:348] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.407140 19826 raft_consensus.cc:374] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:39.407164 19826 raft_consensus.cc:729] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4106f645342b444fb6ed90c80b02bb18, State: Initialized, Role: FOLLOWER
I20260620 14:53:39.407279 19826 consensus_queue.cc:260] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.407557 19826 ts_tablet_manager.cc:1428] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:53:39.407593 19806 heartbeater.cc:503] Master 127.5.118.190:38567 was elected leader, sending a full tablet report...
I20260620 14:53:39.418803 19827 raft_consensus.cc:482] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:39.418936 19827 raft_consensus.cc:504] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.419318 19827 leader_election.cc:288] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805), 61b7957b2eff4649b72bf7295887eeb2 (127.5.118.129:38877)
I20260620 14:53:39.423404 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.419692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:34379 (local address 127.5.118.129:38877)
0620 14:53:39.419735 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.419737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.419745 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:53:39.419805 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.419806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.419814 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.419868 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:39.420033 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.420812 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.422368 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.422991 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.423075 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.423083 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.423261 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.423325 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:53:39.423398 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.419636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:59023 (local address 127.5.118.131:40805)
0620 14:53:39.419717 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.419720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.419730 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:53:39.419760 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.419761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.419770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.419837 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:39.420039 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.420796 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.422353 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.422997 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.423075 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.423083 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.423261 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.423319 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:53:39.423480 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.419515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:40805 (local address 127.5.118.130:59023)
0620 14:53:39.419586 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.419596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.419674 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.419869 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.419870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.419872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:39.419946 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.419948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.420885 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.420886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.422257 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.422260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.423085 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.423086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.423176 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.423178 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.423240 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.423310 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":47}
I20260620 14:53:39.423559 19831 raft_consensus.cc:482] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:39.423662 19831 raft_consensus.cc:504] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.423736 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.419529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:38877 (local address 127.5.118.130:34379)
0620 14:53:39.419702 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.419711 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.419750 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.419878 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.419878 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.419879 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:39.419955 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.419956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.420885 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.420886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.422257 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.422260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.423085 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.423086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.423173 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.423177 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.423240 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.423310 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":109,"thread_start_us":108,"threads_started":1}
I20260620 14:53:39.423839 19760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac37378591b341078262282705ac40de" candidate_uuid: "ea490d5a70a94ba89549a37c83fff72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4106f645342b444fb6ed90c80b02bb18" is_pre_election: true
I20260620 14:53:39.423920 19760 raft_consensus.cc:2454] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea490d5a70a94ba89549a37c83fff72a in term 0.
I20260620 14:53:39.424026 19492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac37378591b341078262282705ac40de" candidate_uuid: "ea490d5a70a94ba89549a37c83fff72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b7957b2eff4649b72bf7295887eeb2" is_pre_election: true
I20260620 14:53:39.424070 19831 leader_election.cc:288] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61b7957b2eff4649b72bf7295887eeb2 (127.5.118.129:38877), ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551)
I20260620 14:53:39.424167 19492 raft_consensus.cc:2454] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea490d5a70a94ba89549a37c83fff72a in term 0.
I20260620 14:53:39.424347 19562 leader_election.cc:302] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4106f645342b444fb6ed90c80b02bb18, ea490d5a70a94ba89549a37c83fff72a; no voters:
I20260620 14:53:39.424643 19827 raft_consensus.cc:2781] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:39.424779 19827 raft_consensus.cc:482] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:39.424860 19827 raft_consensus.cc:3037] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:39.425800 19827 raft_consensus.cc:504] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.425952 19827 leader_election.cc:288] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [CANDIDATE]: Term 1 election: Requested vote from peers 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805), 61b7957b2eff4649b72bf7295887eeb2 (127.5.118.129:38877)
I20260620 14:53:39.426659 19760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac37378591b341078262282705ac40de" candidate_uuid: "ea490d5a70a94ba89549a37c83fff72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4106f645342b444fb6ed90c80b02bb18"
I20260620 14:53:39.426688 19492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac37378591b341078262282705ac40de" candidate_uuid: "ea490d5a70a94ba89549a37c83fff72a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b7957b2eff4649b72bf7295887eeb2"
I20260620 14:53:39.426750 19760 raft_consensus.cc:3037] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:39.426771 19492 raft_consensus.cc:3037] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:39.427668 19760 raft_consensus.cc:2454] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea490d5a70a94ba89549a37c83fff72a in term 1.
I20260620 14:53:39.427767 19492 raft_consensus.cc:2454] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea490d5a70a94ba89549a37c83fff72a in term 1.
I20260620 14:53:39.427840 19562 leader_election.cc:302] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4106f645342b444fb6ed90c80b02bb18, ea490d5a70a94ba89549a37c83fff72a; no voters:
I20260620 14:53:39.427927 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.424221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:38877 (local address 127.5.118.131:59973)
0620 14:53:39.424330 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.424341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.424390 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.424628 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.424629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.424630 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:39.424687 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.424688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.425498 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.425498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.426804 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.426806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.427658 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.427659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.427748 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.427751 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.427798 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.427844 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":98}
I20260620 14:53:39.428014 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.424441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:56527 (local address 127.5.118.130:33551)
0620 14:53:39.424489 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.424491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.424499 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:53:39.424619 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.424620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.424629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.424660 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:39.424872 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.425357 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.426897 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.427432 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.427483 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.427492 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.427786 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.427832 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:53:39.428020 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.424243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:33551 (local address 127.5.118.131:56527)
0620 14:53:39.424399 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.424410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.424534 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.424720 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.424721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.424722 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:39.424793 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.424795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.425463 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.425464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.426779 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.426782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.427597 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.427598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.427690 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.427694 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.427897 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.427941 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":50,"thread_start_us":107,"threads_started":1}
I20260620 14:53:39.428093 19827 raft_consensus.cc:2781] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:39.428298 19827 raft_consensus.cc:686] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 1 LEADER]: Becoming Leader. State: Replica: ea490d5a70a94ba89549a37c83fff72a, State: Running, Role: LEADER
I20260620 14:53:39.428329 19626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac37378591b341078262282705ac40de" candidate_uuid: "4106f645342b444fb6ed90c80b02bb18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea490d5a70a94ba89549a37c83fff72a" is_pre_election: true
I20260620 14:53:39.428390 19827 consensus_queue.cc:237] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.428640 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.424260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:59973 (local address 127.5.118.129:38877)
0620 14:53:39.424299 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.424301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.424518 (+ 217us) server_negotiation.cc:407] Connection header received
0620 14:53:39.424541 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.424542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.424551 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.424589 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:39.424761 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.425392 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.426997 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.427537 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.427614 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.427623 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.428540 (+ 917us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.428586 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:53:39.428809 19492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac37378591b341078262282705ac40de" candidate_uuid: "4106f645342b444fb6ed90c80b02bb18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b7957b2eff4649b72bf7295887eeb2" is_pre_election: true
I20260620 14:53:39.428987 19492 raft_consensus.cc:2379] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4106f645342b444fb6ed90c80b02bb18 in current term 1: Already voted for candidate ea490d5a70a94ba89549a37c83fff72a in this term.
I20260620 14:53:39.429244 19693 leader_election.cc:302] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4106f645342b444fb6ed90c80b02bb18; no voters: 61b7957b2eff4649b72bf7295887eeb2, ea490d5a70a94ba89549a37c83fff72a
I20260620 14:53:39.429383 19831 raft_consensus.cc:2726] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:53:39.429365 19347 catalog_manager.cc:5495] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a reported cstate change: term changed from 0 to 1, leader changed from <none> to ea490d5a70a94ba89549a37c83fff72a (127.5.118.130). New cstate: current_term: 1 leader_uuid: "ea490d5a70a94ba89549a37c83fff72a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } health_report { overall_health: HEALTHY } } }
I20260620 14:53:39.438737 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.436463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.5.118.130:33551)
0620 14:53:39.436509 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.436511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.436523 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:53:39.436614 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.436615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.436630 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.436688 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:39.436807 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.437316 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.438049 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.438230 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.438290 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.438402 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:39.438495 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:39.438550 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.438597 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.438611 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:53:39.440646 19492 raft_consensus.cc:1261] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Refusing update from remote peer ea490d5a70a94ba89549a37c83fff72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:39.440650 19760 raft_consensus.cc:1261] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 1 FOLLOWER]: Refusing update from remote peer ea490d5a70a94ba89549a37c83fff72a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:39.440986 19834 consensus_queue.cc:1035] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:39.441140 19827 consensus_queue.cc:1035] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:39.444059 19838 mvcc.cc:205] Tried to move back new op lower bound from 7298937730825781248 to 7298937730779279360. Current Snapshot: MvccSnapshot[applied={T|T < 7298937730825781248}]
I20260620 14:53:39.444238 19839 mvcc.cc:205] Tried to move back new op lower bound from 7298937730825781248 to 7298937730779279360. Current Snapshot: MvccSnapshot[applied={T|T < 7298937730825781248}]
I20260620 14:53:39.446780 19346 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:39.446882 19346 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:39.447608 19346 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Sending DeleteTablet for 3 replicas of tablet ac37378591b341078262282705ac40de
I20260620 14:53:39.447942 19740 tablet_service.cc:1514] Processing DeleteTablet for tablet ac37378591b341078262282705ac40de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:39 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:39102
I20260620 14:53:39.448020 19472 tablet_service.cc:1514] Processing DeleteTablet for tablet ac37378591b341078262282705ac40de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:39 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:52584
I20260620 14:53:39.448108 19606 tablet_service.cc:1514] Processing DeleteTablet for tablet ac37378591b341078262282705ac40de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:39 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:38394
I20260620 14:53:39.448287 19848 tablet_replica.cc:330] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: stopping tablet replica
I20260620 14:53:39.448329 19847 tablet_replica.cc:330] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18: stopping tablet replica
I20260620 14:53:39.448390 19848 raft_consensus.cc:2229] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:39.448410 19847 raft_consensus.cc:2229] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:39.448388 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 19409
I20260620 14:53:39.448491 19847 raft_consensus.cc:2258] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:39.448491 19848 raft_consensus.cc:2258] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:39.449062 19849 tablet_replica.cc:330] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a: stopping tablet replica
I20260620 14:53:39.449160 19849 raft_consensus.cc:2229] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:53:39.449285 19849 raft_consensus.cc:2258] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:39.449328 19847 ts_tablet_manager.cc:1905] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:39.449684 19849 ts_tablet_manager.cc:1905] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:39.451283 19847 ts_tablet_manager.cc:1918] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:53:39.451294 19849 ts_tablet_manager.cc:1918] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:53:39.451355 19849 log.cc:1196] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/ac37378591b341078262282705ac40de
I20260620 14:53:39.451355 19847 log.cc:1196] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/ac37378591b341078262282705ac40de
I20260620 14:53:39.451661 19849 ts_tablet_manager.cc:1939] T ac37378591b341078262282705ac40de P ea490d5a70a94ba89549a37c83fff72a: Deleting consensus metadata
I20260620 14:53:39.451961 19847 ts_tablet_manager.cc:1939] T ac37378591b341078262282705ac40de P 4106f645342b444fb6ed90c80b02bb18: Deleting consensus metadata
I20260620 14:53:39.452104 19333 catalog_manager.cc:4841] TS ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551): tablet ac37378591b341078262282705ac40de (table test-table [id=14fcf0911d7b473297c3dc32eea26ea6]) successfully deleted
I20260620 14:53:39.452375 19334 catalog_manager.cc:4841] TS 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805): tablet ac37378591b341078262282705ac40de (table test-table [id=14fcf0911d7b473297c3dc32eea26ea6]) successfully deleted
W20260620 14:53:39.456656 19331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:38877 (error 108)
I20260620 14:53:39.457273 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.457127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:38877 (local address 127.0.0.1:52590)
0620 14:53:39.457187 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.457233 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:53:39.457293 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 19543
W20260620 14:53:39.457405 19331 catalog_manager.cc:4584] TS 61b7957b2eff4649b72bf7295887eeb2 (127.5.118.129:38877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac37378591b341078262282705ac40de: Network error: Client connection negotiation failed: client connection to 127.5.118.129:38877: connect: Connection refused (error 111)
I20260620 14:53:39.462489 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 19677
I20260620 14:53:39.467432 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 19315
I20260620 14:53:39.472783 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:38567
--webserver_interface=127.5.118.190
--webserver_port=37045
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:38567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:39.555296 19851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:39.555518 19851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:39.555542 19851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:39.556893 19851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:39.556942 19851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:39.556964 19851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:39.556977 19851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:39.556989 19851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:39.558650 19851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:38567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:38567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=37045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:39.558949 19851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:39.559216 19851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:39.568423 19857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:39.568436 19858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:39.568445 19860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:39.568763 19851 server_base.cc:1034] running on GCE node
I20260620 14:53:39.569034 19851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:39.569389 19851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:39.570601 19851 hybrid_clock.cc:648] HybridClock initialized: now 1781967219570588 us; error 32 us; skew 500 ppm
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:39.578529 19851 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:53:39.579095 19851 webserver.cc:460] Webserver started at http://127.5.118.190:37045/ using document root <none> and password file <none>
I20260620 14:53:39.579365 19851 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:39.579468 19851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:39.581070 19851 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:53:39.581915 19867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.582166 19851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:53:39.582284 19851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "8ab3250e20914d13bcd6b12f55b77d13"
format_stamp: "Formatted at 2026-06-20 14:53:38 on dist-test-slave-2m5f"
server_key: "27363d0b8d43069838dc07c33b95d992"
server_key_iv: "934e5f6a0b6792961c771eac971b435a"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.582746 19851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:39.595438 19851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:39.597478 19851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:39.597647 19851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:39.601889 19851 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:38567
I20260620 14:53:39.601926 19919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:38567 every 8 connection(s)
I20260620 14:53:39.602334 19851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:39.603999 19920 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:53:39.604766 19920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Bootstrap starting.
I20260620 14:53:39.607996 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 19851
I20260620 14:53:39.608412 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:38877
--local_ip_for_outbound_sockets=127.5.118.129
--tserver_master_addrs=127.5.118.190:38567
--webserver_port=34235
--webserver_interface=127.5.118.129
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:39.613864 19920 log.cc:824] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:39.615990 19920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:53:39.616372 19920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Bootstrap complete.
I20260620 14:53:39.618706 19920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:39.618939 19920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab3250e20914d13bcd6b12f55b77d13, State: Initialized, Role: FOLLOWER
I20260620 14:53:39.619115 19920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:39.619189 19920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:39.619243 19920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:39.619292 19920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:53:39.620487 19920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:39.620617 19920 leader_election.cc:302] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ab3250e20914d13bcd6b12f55b77d13; no voters:
I20260620 14:53:39.620816 19920 leader_election.cc:288] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:53:39.620857 19924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:53:39.620970 19924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 2 LEADER]: Becoming Leader. State: Replica: 8ab3250e20914d13bcd6b12f55b77d13, State: Running, Role: LEADER
I20260620 14:53:39.621079 19924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:39.621109 19920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:39.621515 19926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ab3250e20914d13bcd6b12f55b77d13. Latest consensus state: current_term: 2 leader_uuid: "8ab3250e20914d13bcd6b12f55b77d13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } } }
I20260620 14:53:39.621609 19926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:39.621836 19928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:39.622632 19928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:39.622728 19928 catalog_manager.cc:1259] Loaded cluster ID: 67e63a1ee40d447e90ed4fc9327f7fec
I20260620 14:53:39.622979 19928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:39.623064 19925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8ab3250e20914d13bcd6b12f55b77d13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } } }
I20260620 14:53:39.623200 19925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:39.623245 19928 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 14:53:39.627942 19928 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 14:53:39.630606 19928 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:39.630839 19928 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Loaded TSK: 0
I20260620 14:53:39.631070 19928 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:53:39.694079 19922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:39.694337 19922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:39.694379 19922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:39.695645 19922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:39.695725 19922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:39.695772 19922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:53:39.697436 19922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:38877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=34235
--tserver_master_addrs=127.5.118.190:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:39.697749 19922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:39.698004 19922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:39.703308 19950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:39.703397 19947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:39.703646 19922 server_base.cc:1034] running on GCE node
W20260620 14:53:39.703651 19948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:39.703986 19922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:39.704329 19922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:39.705518 19922 hybrid_clock.cc:648] HybridClock initialized: now 1781967219705502 us; error 34 us; skew 500 ppm
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:39.708793 19922 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:53:39.709348 19922 webserver.cc:460] Webserver started at http://127.5.118.129:34235/ using document root <none> and password file <none>
I20260620 14:53:39.709594 19922 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:39.709685 19922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:39.711174 19922 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:39.711953 19957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.712137 19922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:39.712268 19922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "61b7957b2eff4649b72bf7295887eeb2"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "57e0fe9d0b4f270509aedead5b00c12c"
server_key_iv: "5bbbbe5443e90519b679a0477e5f7a4e"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.712652 19922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:39.724537 19922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:39.726963 19922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:39.727084 19922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:39.727356 19922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:39.727829 19964 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:53:39.729876 19922 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:53:39.730000 19922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:53:39.730067 19922 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:53:39.730712 19922 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:53:39.730768 19922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:53:39.730850 19964 tablet_bootstrap.cc:492] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Bootstrap starting.
I20260620 14:53:39.738305 19922 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:38877
I20260620 14:53:39.738713 19922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:39.740410 20071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:38877 every 8 connection(s)
I20260620 14:53:39.742385 19964 log.cc:824] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:39.744290 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 19922
I20260620 14:53:39.744609 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:33551
--local_ip_for_outbound_sockets=127.5.118.130
--tserver_master_addrs=127.5.118.190:38567
--webserver_port=45019
--webserver_interface=127.5.118.130
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:39.745503 19964 tablet_bootstrap.cc:492] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:53:39.745812 19964 tablet_bootstrap.cc:492] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Bootstrap complete.
I20260620 14:53:39.746871 19964 ts_tablet_manager.cc:1397] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260620 14:53:39.748383 19964 raft_consensus.cc:348] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.748713 19964 raft_consensus.cc:729] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b7957b2eff4649b72bf7295887eeb2, State: Initialized, Role: FOLLOWER
I20260620 14:53:39.748878 19964 consensus_queue.cc:260] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } }
I20260620 14:53:39.749397 19964 ts_tablet_manager.cc:1428] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:39.755324 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.739474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:46029)
0620 14:53:39.739686 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.739732 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.740316 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.743323 (+ 3007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.743328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.743723 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:39.743991 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.744000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.746696 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.746697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.747966 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.747968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.749139 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.749140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.749295 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.750018 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:39.750029 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:39.751491 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:39.753575 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:39.753585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:39.753587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:39.753830 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:39.754134 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:39.754135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:39.754135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:39.754254 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:39.754654 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:39.754661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:39.754810 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.754928 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.755114 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":127,"threads_started":1}
I20260620 14:53:39.755352 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.739556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:46029 (local address 127.5.118.190:38567)
0620 14:53:39.742808 (+ 3252us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.742820 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.742874 (+ 54us) server_negotiation.cc:407] Connection header received
0620 14:53:39.743029 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.743036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.743102 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.743278 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:39.745003 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.746566 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.748082 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.748745 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.748833 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.751637 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:39.751648 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:39.751649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:39.751661 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:39.753457 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.753940 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.753943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.753944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.754045 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.754334 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.754335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.754336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.754572 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:39.754676 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.754983 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.755115 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3187,"thread_start_us":105,"threads_started":1}
I20260620 14:53:39.756523 20072 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38567
I20260620 14:53:39.756628 20072 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:39.756815 20072 heartbeater.cc:511] Master 127.5.118.190:38567 requested a full tablet report, sending...
I20260620 14:53:39.757673 19884 ts_manager.cc:194] Registered new tserver with Master: 61b7957b2eff4649b72bf7295887eeb2 (127.5.118.129:38877)
I20260620 14:53:39.758917 19884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:46029
I20260620 14:53:39.759843 20072 heartbeater.cc:503] Master 127.5.118.190:38567 was elected leader, sending a full tablet report...
I20260620 14:53:39.763661 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.758577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.5.118.129:38877)
0620 14:53:39.760533 (+ 1956us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.760537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.760544 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:53:39.760569 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.760570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.760582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.760677 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:39.760826 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.761511 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.762528 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.763213 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.763310 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.763371 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.763545 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.763585 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1929,"thread_start_us":81,"threads_started":1}
I20260620 14:53:39.763660 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.758322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:38877 (local address 127.0.0.1:52600)
0620 14:53:39.760299 (+ 1977us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.760339 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.760404 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.760682 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.760685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.760687 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:39.760753 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.760755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.761600 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.761601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.762424 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.762426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.763313 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.763315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.763408 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.763422 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.763526 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.763585 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1943,"thread_start_us":93,"threads_started":1}
I20260620 14:53:39.764320 20004 tablet_service.cc:1514] Processing DeleteTablet for tablet ac37378591b341078262282705ac40de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:39 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:52600
I20260620 14:53:39.764946 20081 tablet_replica.cc:330] stopping tablet replica
I20260620 14:53:39.765079 20081 raft_consensus.cc:2229] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:39.765151 20081 raft_consensus.cc:2258] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:39.765467 20081 ts_tablet_manager.cc:1905] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:39.767652 20081 ts_tablet_manager.cc:1918] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:53:39.767747 20081 log.cc:1196] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal/wals/ac37378591b341078262282705ac40de
I20260620 14:53:39.767997 20081 ts_tablet_manager.cc:1939] T ac37378591b341078262282705ac40de P 61b7957b2eff4649b72bf7295887eeb2: Deleting consensus metadata
I20260620 14:53:39.768590 19868 catalog_manager.cc:4841] TS 61b7957b2eff4649b72bf7295887eeb2 (127.5.118.129:38877): tablet ac37378591b341078262282705ac40de (table test-table [id=14fcf0911d7b473297c3dc32eea26ea6]) successfully deleted
W20260620 14:53:39.836494 20077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:39.836710 20077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:39.836735 20077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:39.837884 20077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:39.837924 20077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:39.837951 20077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:53:39.839365 20077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:33551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=45019
--tserver_master_addrs=127.5.118.190:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:39.839573 20077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:39.839756 20077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:39.845058 20086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:39.845208 20077 server_base.cc:1034] running on GCE node
W20260620 14:53:39.845048 20089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:39.845048 20087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:39.845556 20077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:39.845832 20077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:39.847008 20077 hybrid_clock.cc:648] HybridClock initialized: now 1781967219846985 us; error 38 us; skew 500 ppm
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:39.849889 20077 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:53:39.850385 20077 webserver.cc:460] Webserver started at http://127.5.118.130:45019/ using document root <none> and password file <none>
I20260620 14:53:39.850586 20077 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:39.850642 20077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:39.852233 20077 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:53:39.852838 20096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.853070 20077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:53:39.853170 20077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "ea490d5a70a94ba89549a37c83fff72a"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "f345e10e929e568e4d3d4e3667710fc0"
server_key_iv: "e8d8409166edd4837cf22a82198e85b9"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.853528 20077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:39.865558 20077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:39.867997 20077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:39.868116 20077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:39.868384 20077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:39.868654 20077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:39.868687 20077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.868709 20077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:39.868721 20077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.874850 20077 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:33551
I20260620 14:53:39.875355 20077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:39.875330 20209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:33551 every 8 connection(s)
I20260620 14:53:39.881911 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 20077
I20260620 14:53:39.882192 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:40805
--local_ip_for_outbound_sockets=127.5.118.131
--tserver_master_addrs=127.5.118.190:38567
--webserver_port=34963
--webserver_interface=127.5.118.131
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:39.886416 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.876491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:42207 (local address 127.5.118.190:38567)
0620 14:53:39.876560 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:39.876562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:39.876786 (+ 224us) server_negotiation.cc:407] Connection header received
0620 14:53:39.877442 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:39.877444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:39.877460 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:39.877519 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:39.878521 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.879054 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.880098 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.880419 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.880509 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.882931 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:39.882934 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:39.882934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:39.882943 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:39.884650 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.885203 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.885205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.885205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.885265 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:39.885603 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:39.885604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:39.885605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:39.885742 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:39.885810 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:39.886196 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 14:53:39.886322 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 14:53:39.886709 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:39.876494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:42207)
0620 14:53:39.876658 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:39.876696 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:39.877298 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:39.877598 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:39.877602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:39.878197 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:39.878400 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.878405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.879159 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.879160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.879996 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:39.879999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:39.880523 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:39.880524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:39.880650 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:39.881233 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:39.881242 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:39.882814 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:39.884755 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:39.884764 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:39.884766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:39.885103 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:39.885336 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:39.885337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:39.885337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:39.885495 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:39.885822 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:39.885829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:39.886036 (+ 207us) client_negotiation.cc:769] Sending connection context
0620 14:53:39.886163 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:53:39.886376 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260620 14:53:39.887275 20210 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38567
I20260620 14:53:39.887393 20210 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:39.887624 20210 heartbeater.cc:511] Master 127.5.118.190:38567 requested a full tablet report, sending...
I20260620 14:53:39.888047 19884 ts_manager.cc:194] Registered new tserver with Master: ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551)
I20260620 14:53:39.888473 19884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:42207
W20260620 14:53:39.969267 20213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:39.969537 20213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:39.969583 20213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:39.970865 20213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:39.970945 20213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:39.970992 20213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:53:39.972545 20213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:40805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=34963
--tserver_master_addrs=127.5.118.190:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:39.972838 20213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:39.973096 20213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:39.978595 20221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:39.978583 20218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:39.978583 20219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:39.978847 20213 server_base.cc:1034] running on GCE node
I20260620 14:53:39.979166 20213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:39.979452 20213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:39.980618 20213 hybrid_clock.cc:648] HybridClock initialized: now 1781967219980619 us; error 52 us; skew 500 ppm
Jun 20 14:53:39 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:39.983366 20213 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:53:39.983856 20213 webserver.cc:460] Webserver started at http://127.5.118.131:34963/ using document root <none> and password file <none>
I20260620 14:53:39.984130 20213 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:39.984246 20213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:39.985790 20213 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:39.986569 20228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:39.986800 20213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:39.986912 20213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "4106f645342b444fb6ed90c80b02bb18"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "8441e4645b34b62fb9af2419e5706158"
server_key_iv: "983e0ed7073a2423f29208b1633845e7"
server_key_version: "encryptionkey@0"
I20260620 14:53:39.987313 20213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:39.997195 20213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:39.999230 20213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:39.999395 20213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:39.999637 20213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:39.999943 20213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:39.999997 20213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:40.000042 20213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:40.000074 20213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:40.006108 20213 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:40805
I20260620 14:53:40.006151 20341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:40805 every 8 connection(s)
I20260620 14:53:40.006508 20213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:40.008311 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 20213
Jun 20 14:53:40 dist-test-slave-2m5f krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:40.016075 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.008717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.5.118.190:38567)
0620 14:53:40.008962 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.008963 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.008969 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:53:40.008987 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.008988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.008997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.009066 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:40.009717 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.010244 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.010991 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.011312 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.011374 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.012829 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:40.012831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:40.012831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:40.012838 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:40.014480 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:40.015094 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:40.015095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:40.015095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:40.015154 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:40.015395 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:40.015396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:40.015396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:40.015511 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:40.015644 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.015829 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.015950 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":67,"threads_started":1}
I20260620 14:53:40.017606 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.007268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:57885 (local address 127.5.118.190:38567)
0620 14:53:40.007343 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.007346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.007616 (+ 270us) server_negotiation.cc:407] Connection header received
0620 14:53:40.008154 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.008155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.008167 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.008300 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:40.009461 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.010133 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.011306 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.011636 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.011708 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.014296 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:40.014298 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:40.014298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:40.014306 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:40.015982 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:40.016751 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:40.016752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:40.016752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:40.016812 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:40.017065 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:40.017065 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:40.017065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:40.017165 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:40.017215 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.017463 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.017547 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":40}
I20260620 14:53:40.017805 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.007277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:57885)
0620 14:53:40.007478 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.007517 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:40.008027 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:40.008415 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:40.008421 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:40.009155 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:40.009359 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.009365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.010236 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.010237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.011215 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.011218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.011733 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.011734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.011888 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.012480 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:40.012489 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:40.014176 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:40.016161 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:40.016169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:40.016171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:40.016531 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:40.016880 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:40.016881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:40.016881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:40.017002 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:40.017222 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:40.017227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:40.017349 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 14:53:40.017437 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 14:53:40.017562 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":88,"threads_started":1}
I20260620 14:53:40.018309 20342 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38567
I20260620 14:53:40.018421 20342 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:40.018610 20342 heartbeater.cc:511] Master 127.5.118.190:38567 requested a full tablet report, sending...
I20260620 14:53:40.019004 19884 ts_manager.cc:194] Registered new tserver with Master: 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805)
I20260620 14:53:40.019377 19884 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:57885
I20260620 14:53:40.028091 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:53:40.031317 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.028575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.5.118.190:38567)
0620 14:53:40.028638 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.028640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.028760 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:53:40.028805 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.028806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.028852 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.028924 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:40.029117 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.029603 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.030486 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.030688 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.030741 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.030952 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:40.031069 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:40.031135 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.031228 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.031247 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:53:40.031656 19884 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:53:40.033350 19884 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:40.035612 20004 tablet_service.cc:1467] Processing CreateTablet for tablet a06d14febdbe49cb945178d6270ba7cb (DEFAULT_TABLE table=test-table [id=aeef40668ddb461e919f760154600275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:40.035849 20004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06d14febdbe49cb945178d6270ba7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:40.037573 19964 tablet_bootstrap.cc:492] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: Bootstrap starting.
I20260620 14:53:40.037911 19964 tablet_bootstrap.cc:654] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:40.038527 19964 tablet_bootstrap.cc:492] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: No bootstrap required, opened a new log
I20260620 14:53:40.038592 19964 ts_tablet_manager.cc:1397] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:40.038702 19964 raft_consensus.cc:348] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:40.038769 19964 raft_consensus.cc:374] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:40.038818 19964 raft_consensus.cc:729] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b7957b2eff4649b72bf7295887eeb2, State: Initialized, Role: FOLLOWER
I20260620 14:53:40.038865 19964 consensus_queue.cc:260] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:40.038955 19964 ts_tablet_manager.cc:1428] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:40.039430 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.035357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:40805 (local address 127.0.0.1:39116)
0620 14:53:40.035691 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.035700 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:40.035748 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:40.036078 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:40.036078 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:40.036080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:40.036117 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.036118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.037173 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.037174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.038207 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.038211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.039170 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.039171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.039238 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.039243 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:53:40.039302 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:53:40.039346 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":155,"thread_start_us":121,"threads_started":1}
I20260620 14:53:40.039619 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.035356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:33551 (local address 127.0.0.1:38418)
0620 14:53:40.035411 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.035423 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:40.035462 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:40.035875 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:40.035876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:40.035877 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:40.035931 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.035933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.037260 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.037261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.038317 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.038321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.039393 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.039394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.039450 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.039453 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:53:40.039501 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:53:40.039566 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 14:53:40.039811 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.035484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.5.118.131:40805)
0620 14:53:40.035907 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.035913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.035923 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:53:40.035963 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.035965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.035984 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.036232 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:40.036314 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.037068 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.038309 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.039079 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.039155 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.039223 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.039685 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.039736 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":115,"threads_started":1}
I20260620 14:53:40.039852 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.035456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38418 (local address 127.5.118.130:33551)
0620 14:53:40.035705 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.035710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.035722 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:53:40.035766 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.035768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.035785 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.035879 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:40.036000 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.037128 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.038422 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.039240 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.039310 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.039363 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.039726 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.039768 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":136,"threads_started":1}
I20260620 14:53:40.040534 20144 tablet_service.cc:1467] Processing CreateTablet for tablet a06d14febdbe49cb945178d6270ba7cb (DEFAULT_TABLE table=test-table [id=aeef40668ddb461e919f760154600275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:40.040527 20276 tablet_service.cc:1467] Processing CreateTablet for tablet a06d14febdbe49cb945178d6270ba7cb (DEFAULT_TABLE table=test-table [id=aeef40668ddb461e919f760154600275]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:40.040867 20276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06d14febdbe49cb945178d6270ba7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:40.040877 20144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a06d14febdbe49cb945178d6270ba7cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:40.043665 20352 tablet_bootstrap.cc:492] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a: Bootstrap starting.
I20260620 14:53:40.043689 20353 tablet_bootstrap.cc:492] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18: Bootstrap starting.
I20260620 14:53:40.044359 20352 tablet_bootstrap.cc:654] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:40.044371 20353 tablet_bootstrap.cc:654] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:40.044873 20353 log.cc:824] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:40.044898 20352 log.cc:824] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:40.045588 20353 tablet_bootstrap.cc:492] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18: No bootstrap required, opened a new log
I20260620 14:53:40.045644 20353 ts_tablet_manager.cc:1397] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:40.045637 20352 tablet_bootstrap.cc:492] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a: No bootstrap required, opened a new log
I20260620 14:53:40.045698 20352 ts_tablet_manager.cc:1397] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:40.046974 20353 raft_consensus.cc:348] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:40.047122 20353 raft_consensus.cc:374] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:40.047147 20353 raft_consensus.cc:729] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4106f645342b444fb6ed90c80b02bb18, State: Initialized, Role: FOLLOWER
I20260620 14:53:40.047013 20352 raft_consensus.cc:348] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:40.047183 20352 raft_consensus.cc:374] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:40.047230 20352 raft_consensus.cc:729] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea490d5a70a94ba89549a37c83fff72a, State: Initialized, Role: FOLLOWER
I20260620 14:53:40.047298 20353 consensus_queue.cc:260] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:40.047341 20352 consensus_queue.cc:260] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:40.047633 20352 ts_tablet_manager.cc:1428] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:40.047621 20353 ts_tablet_manager.cc:1428] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:40.047632 20342 heartbeater.cc:503] Master 127.5.118.190:38567 was elected leader, sending a full tablet report...
I20260620 14:53:40.048058 20210 heartbeater.cc:503] Master 127.5.118.190:38567 was elected leader, sending a full tablet report...
I20260620 14:53:40.071642 20356 raft_consensus.cc:482] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:40.071796 20356 raft_consensus.cc:504] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:40.072263 20356 leader_election.cc:288] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551), 61b7957b2eff4649b72bf7295887eeb2 (127.5.118.129:38877)
I20260620 14:53:40.075309 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.072724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:47307 (local address 127.5.118.129:38877)
0620 14:53:40.072776 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.072778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.072788 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:53:40.072882 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.072883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.072892 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.072943 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:40.073155 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.073600 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.074530 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.074914 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.075006 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.075017 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.075187 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.075234 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:53:40.075337 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.072551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:38877 (local address 127.5.118.131:47307)
0620 14:53:40.072745 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.072756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:40.072799 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:40.072969 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:40.072970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:40.072972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:40.073043 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.073045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.073690 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.073691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.074436 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.074438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.075022 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.075023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.075112 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.075115 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:53:40.075181 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:53:40.075221 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":3,"thread_start_us":117,"threads_started":1}
I20260620 14:53:40.075366 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.072563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:58265 (local address 127.5.118.130:33551)
0620 14:53:40.072612 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.072614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.072624 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:53:40.072653 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.072654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.072664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.072727 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:40.073044 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.073533 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.074540 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.074939 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.075026 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.075035 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.075231 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.075274 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:53:40.075399 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.072489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:33551 (local address 127.5.118.131:58265)
0620 14:53:40.072543 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.072555 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:40.072595 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:40.072789 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:40.072790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:40.072792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:40.072867 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.072869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.073622 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.073623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.074441 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:40.074443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.075023 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.075024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:40.075147 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.075150 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:53:40.075225 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 14:53:40.075271 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":40}
I20260620 14:53:40.075742 20164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06d14febdbe49cb945178d6270ba7cb" candidate_uuid: "4106f645342b444fb6ed90c80b02bb18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea490d5a70a94ba89549a37c83fff72a" is_pre_election: true
I20260620 14:53:40.075953 20164 raft_consensus.cc:2454] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4106f645342b444fb6ed90c80b02bb18 in term 0.
I20260620 14:53:40.075927 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06d14febdbe49cb945178d6270ba7cb" candidate_uuid: "4106f645342b444fb6ed90c80b02bb18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b7957b2eff4649b72bf7295887eeb2" is_pre_election: true
I20260620 14:53:40.076066 20026 raft_consensus.cc:2454] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4106f645342b444fb6ed90c80b02bb18 in term 0.
I20260620 14:53:40.076246 20231 leader_election.cc:302] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4106f645342b444fb6ed90c80b02bb18, ea490d5a70a94ba89549a37c83fff72a; no voters:
I20260620 14:53:40.076396 20356 raft_consensus.cc:2781] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:40.076478 20356 raft_consensus.cc:482] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:40.076500 20356 raft_consensus.cc:3037] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:40.077160 20356 raft_consensus.cc:504] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:40.077317 20356 leader_election.cc:288] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [CANDIDATE]: Term 1 election: Requested vote from peers ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551), 61b7957b2eff4649b72bf7295887eeb2 (127.5.118.129:38877)
I20260620 14:53:40.077481 20164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06d14febdbe49cb945178d6270ba7cb" candidate_uuid: "4106f645342b444fb6ed90c80b02bb18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea490d5a70a94ba89549a37c83fff72a"
I20260620 14:53:40.077490 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a06d14febdbe49cb945178d6270ba7cb" candidate_uuid: "4106f645342b444fb6ed90c80b02bb18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61b7957b2eff4649b72bf7295887eeb2"
I20260620 14:53:40.077574 20026 raft_consensus.cc:3037] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:40.077584 20164 raft_consensus.cc:3037] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:40.078480 20026 raft_consensus.cc:2454] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4106f645342b444fb6ed90c80b02bb18 in term 1.
I20260620 14:53:40.078480 20164 raft_consensus.cc:2454] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4106f645342b444fb6ed90c80b02bb18 in term 1.
I20260620 14:53:40.078708 20231 leader_election.cc:302] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4106f645342b444fb6ed90c80b02bb18, ea490d5a70a94ba89549a37c83fff72a; no voters:
I20260620 14:53:40.078838 20356 raft_consensus.cc:2781] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:40.079031 20356 raft_consensus.cc:686] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 1 LEADER]: Becoming Leader. State: Replica: 4106f645342b444fb6ed90c80b02bb18, State: Running, Role: LEADER
I20260620 14:53:40.079135 20356 consensus_queue.cc:237] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:40.080066 19883 catalog_manager.cc:5495] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131). New cstate: current_term: 1 leader_uuid: "4106f645342b444fb6ed90c80b02bb18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } health_report { overall_health: UNKNOWN } } }
I20260620 14:53:40.087049 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.084601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.5.118.131:40805)
0620 14:53:40.084666 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:40.084668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:40.084684 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:53:40.084712 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:40.084713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:40.084728 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:40.084790 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:40.084921 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.085483 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.086319 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:40.086515 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:40.086587 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:40.086711 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:40.086810 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:40.086871 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:40.086904 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 14:53:40.086921 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:53:40.088901 20026 raft_consensus.cc:1261] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Refusing update from remote peer 4106f645342b444fb6ed90c80b02bb18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:40.089269 20164 raft_consensus.cc:1261] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a [term 1 FOLLOWER]: Refusing update from remote peer 4106f645342b444fb6ed90c80b02bb18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:40.089299 20359 consensus_queue.cc:1035] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:40.089640 20359 consensus_queue.cc:1035] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:40.092315 20365 mvcc.cc:205] Tried to move back new op lower bound from 7298937733481635840 to 7298937733445083136. Current Snapshot: MvccSnapshot[applied={T|T < 7298937733481635840}]
I20260620 14:53:40.093125 20363 mvcc.cc:205] Tried to move back new op lower bound from 7298937733481635840 to 7298937733445083136. Current Snapshot: MvccSnapshot[applied={T|T < 7298937733481635840 or (T in {7298937733481635840})}]
I20260620 14:53:40.094743 19882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:40.094827 19882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:40.095521 19882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Sending DeleteTablet for 3 replicas of tablet a06d14febdbe49cb945178d6270ba7cb
I20260620 14:53:40.095865 20004 tablet_service.cc:1514] Processing DeleteTablet for tablet a06d14febdbe49cb945178d6270ba7cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:40 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:52600
I20260620 14:53:40.095868 20144 tablet_service.cc:1514] Processing DeleteTablet for tablet a06d14febdbe49cb945178d6270ba7cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:40 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:38418
I20260620 14:53:40.095866 20276 tablet_service.cc:1514] Processing DeleteTablet for tablet a06d14febdbe49cb945178d6270ba7cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:40 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:39116
I20260620 14:53:40.095960 20081 tablet_replica.cc:330] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: stopping tablet replica
I20260620 14:53:40.096011 20081 raft_consensus.cc:2229] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:40.096086 20081 raft_consensus.cc:2258] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:40.096155 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 19922
I20260620 14:53:40.096335 20372 tablet_replica.cc:330] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a: stopping tablet replica
I20260620 14:53:40.096447 20081 ts_tablet_manager.cc:1905] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:40.096460 20373 tablet_replica.cc:330] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18: stopping tablet replica
I20260620 14:53:40.096638 20373 raft_consensus.cc:2229] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:53:40.096771 20372 raft_consensus.cc:2229] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:40.096889 20372 raft_consensus.cc:2258] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:40.097086 20373 raft_consensus.cc:2258] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:40.097381 20372 ts_tablet_manager.cc:1905] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:40.098438 20373 ts_tablet_manager.cc:1905] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:40.099510 20372 ts_tablet_manager.cc:1918] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:53:40.099593 20372 log.cc:1196] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/a06d14febdbe49cb945178d6270ba7cb
I20260620 14:53:40.099877 20372 ts_tablet_manager.cc:1939] T a06d14febdbe49cb945178d6270ba7cb P ea490d5a70a94ba89549a37c83fff72a: Deleting consensus metadata
I20260620 14:53:40.100584 20373 ts_tablet_manager.cc:1918] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:53:40.100666 20373 log.cc:1196] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/a06d14febdbe49cb945178d6270ba7cb
I20260620 14:53:40.101032 20373 ts_tablet_manager.cc:1939] T a06d14febdbe49cb945178d6270ba7cb P 4106f645342b444fb6ed90c80b02bb18: Deleting consensus metadata
I20260620 14:53:40.101502 19871 catalog_manager.cc:4841] TS 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805): tablet a06d14febdbe49cb945178d6270ba7cb (table test-table [id=aeef40668ddb461e919f760154600275]) successfully deleted
W20260620 14:53:40.103204 19868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:38877 (error 108)
I20260620 14:53:40.103384 19870 catalog_manager.cc:4841] TS ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551): tablet a06d14febdbe49cb945178d6270ba7cb (table test-table [id=aeef40668ddb461e919f760154600275]) successfully deleted
I20260620 14:53:40.103935 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 20077
I20260620 14:53:40.108771 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.108602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:38877 (local address 127.0.0.1:52602)
0620 14:53:40.108670 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.108730 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:38877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:53:40.109159 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 20213
W20260620 14:53:40.108932 19868 catalog_manager.cc:4584] TS 61b7957b2eff4649b72bf7295887eeb2 (127.5.118.129:38877): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a06d14febdbe49cb945178d6270ba7cb: Network error: Client connection negotiation failed: client connection to 127.5.118.129:38877: connect: Connection refused (error 111)
I20260620 14:53:40.114065 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 19851
I20260620 14:53:40.119467 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:38567
--webserver_interface=127.5.118.190
--webserver_port=37045
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:38567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:40.206139 20375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:40.206360 20375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:40.206384 20375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:40.207602 20375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:40.207650 20375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:40.207664 20375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:40.207677 20375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:40.207734 20375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:40.209318 20375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:38567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:38567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=37045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:40.209607 20375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:40.209853 20375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:40.214805 20380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:40.214834 20375 server_base.cc:1034] running on GCE node
W20260620 14:53:40.214805 20381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:40.214862 20383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:40.215348 20375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:40.215659 20375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:40.216828 20375 hybrid_clock.cc:648] HybridClock initialized: now 1781967220216815 us; error 45 us; skew 500 ppm
Jun 20 14:53:40 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:40.219278 20375 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:53:40.219749 20375 webserver.cc:460] Webserver started at http://127.5.118.190:37045/ using document root <none> and password file <none>
I20260620 14:53:40.219939 20375 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:40.219982 20375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:40.221419 20375 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:40.222138 20390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:40.222316 20375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:40.222400 20375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "8ab3250e20914d13bcd6b12f55b77d13"
format_stamp: "Formatted at 2026-06-20 14:53:38 on dist-test-slave-2m5f"
server_key: "27363d0b8d43069838dc07c33b95d992"
server_key_iv: "934e5f6a0b6792961c771eac971b435a"
server_key_version: "encryptionkey@0"
I20260620 14:53:40.222754 20375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:40.237339 20375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:40.239773 20375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:40.239899 20375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:40.243848 20375 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:38567
I20260620 14:53:40.243907 20442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:38567 every 8 connection(s)
I20260620 14:53:40.244310 20375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:40.244545 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 20375
I20260620 14:53:40.244849 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:38877
--local_ip_for_outbound_sockets=127.5.118.129
--tserver_master_addrs=127.5.118.190:38567
--webserver_port=34235
--webserver_interface=127.5.118.129
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:40.246301 20443 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:53:40.247335 20443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Bootstrap starting.
I20260620 14:53:40.255868 20443 log.cc:824] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:40.258571 20443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:53:40.258955 20443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13: Bootstrap complete.
I20260620 14:53:40.261379 20443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:40.261623 20443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ab3250e20914d13bcd6b12f55b77d13, State: Initialized, Role: FOLLOWER
I20260620 14:53:40.261813 20443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:40.261884 20443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:40.261938 20443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:40.261996 20443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:53:40.263083 20443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:40.263176 20443 leader_election.cc:302] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ab3250e20914d13bcd6b12f55b77d13; no voters:
I20260620 14:53:40.263417 20443 leader_election.cc:288] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:53:40.263463 20447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:53:40.263589 20447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [term 3 LEADER]: Becoming Leader. State: Replica: 8ab3250e20914d13bcd6b12f55b77d13, State: Running, Role: LEADER
I20260620 14:53:40.263696 20443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:40.263710 20447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } }
I20260620 14:53:40.264086 20448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8ab3250e20914d13bcd6b12f55b77d13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } } }
I20260620 14:53:40.264091 20447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ab3250e20914d13bcd6b12f55b77d13. Latest consensus state: current_term: 3 leader_uuid: "8ab3250e20914d13bcd6b12f55b77d13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ab3250e20914d13bcd6b12f55b77d13" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38567 } } }
I20260620 14:53:40.264321 20448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:40.264343 20447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ab3250e20914d13bcd6b12f55b77d13 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:40.264842 20456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:40.265697 20456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:40.265825 20456 catalog_manager.cc:1259] Loaded cluster ID: 67e63a1ee40d447e90ed4fc9327f7fec
I20260620 14:53:40.265857 20456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 14:53:40.266234 20456 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f24f94c29ad google::LogMessage::Fail() at ??:0
@ 0x7f24f94c588a google::LogMessage::SendToLog() at ??:0
@ 0x7f24f94c24de google::LogMessage::Flush() at ??:0
@ 0x7f24f94c3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f25013e30f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f25013ec8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f24fb3900d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f24fb385954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f24fc1e96db start_thread at ??:0
@ 0x7f24f7fc061f clone at ??:0
W20260620 14:53:40.341310 20445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:40.341563 20445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:40.341603 20445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:40.342810 20445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:40.342881 20445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:40.342927 20445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:53:40.344552 20445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:38877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=34235
--tserver_master_addrs=127.5.118.190:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:40.344839 20445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:40.345120 20445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:40.350421 20480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:40.350462 20479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:40.350561 20445 server_base.cc:1034] running on GCE node
W20260620 14:53:40.350670 20482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:40.350871 20445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:40.351173 20445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:40.352404 20445 hybrid_clock.cc:648] HybridClock initialized: now 1781967220352384 us; error 38 us; skew 500 ppm
Jun 20 14:53:40 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:40.356860 20445 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:53:40.357761 20445 webserver.cc:460] Webserver started at http://127.5.118.129:34235/ using document root <none> and password file <none>
I20260620 14:53:40.358098 20445 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:40.358172 20445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:40.360775 20445 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:53:40.362030 20489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:40.362274 20445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:40.362385 20445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "61b7957b2eff4649b72bf7295887eeb2"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "57e0fe9d0b4f270509aedead5b00c12c"
server_key_iv: "5bbbbe5443e90519b679a0477e5f7a4e"
server_key_version: "encryptionkey@0"
I20260620 14:53:40.363174 20445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:40.382448 20445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:40.385478 20445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:40.385685 20445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:40.385965 20445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:40.386521 20496 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:53:40.388686 20445 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:53:40.388756 20445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:53:40.388794 20445 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:53:40.389611 20445 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:53:40.389662 20445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:53:40.389860 20496 tablet_bootstrap.cc:492] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: Bootstrap starting.
I20260620 14:53:40.392076 20496 log.cc:824] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:40.395754 20496 tablet_bootstrap.cc:492] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:53:40.396215 20496 tablet_bootstrap.cc:492] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: Bootstrap complete.
I20260620 14:53:40.396451 20496 ts_tablet_manager.cc:1397] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260620 14:53:40.397513 20445 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:38877
I20260620 14:53:40.397591 20604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:38877 every 8 connection(s)
I20260620 14:53:40.397918 20445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:53:40.399567 20496 raft_consensus.cc:348] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:40.399986 20496 raft_consensus.cc:729] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b7957b2eff4649b72bf7295887eeb2, State: Initialized, Role: FOLLOWER
I20260620 14:53:40.400141 20496 consensus_queue.cc:260] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:40.400413 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 20445
I20260620 14:53:40.400553 20496 ts_tablet_manager.cc:1428] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:53:40.400731 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:33551
--local_ip_for_outbound_sockets=127.5.118.130
--tserver_master_addrs=127.5.118.190:38567
--webserver_port=45019
--webserver_interface=127.5.118.130
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:40.500420 20609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:40.500689 20609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:40.500732 20609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:40.502018 20609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:40.502100 20609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:40.502152 20609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
W20260620 14:53:40.503945 20607 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:40.398631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:47835)
0620 14:53:40.398832 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.398885 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:40.399572 (+ 687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:40.503717 (+104145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
W20260620 14:53:40.504254 20605 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:38567 (0 consecutive failures): Network error: Failed to ping master at 127.5.118.190:38567: Client connection negotiation failed: client connection to 127.5.118.190:38567: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:53:40.504314 20609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:33551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=45019
--tserver_master_addrs=127.5.118.190:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:40.504606 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.504485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:56445)
0620 14:53:40.504537 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.504579 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:53:40.504634 20609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:40.504915 20609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:53:40.504926 20607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.504822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:33331)
0620 14:53:40.504868 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.504899 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:53:40.505041 20605 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:53:40.510533 20614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:40.510545 20615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:40.510589 20617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:40.510717 20609 server_base.cc:1034] running on GCE node
I20260620 14:53:40.511059 20609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:40.511350 20609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:40.512567 20609 hybrid_clock.cc:648] HybridClock initialized: now 1781967220512561 us; error 58 us; skew 500 ppm
Jun 20 14:53:40 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:40.515153 20609 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:53:40.515659 20609 webserver.cc:460] Webserver started at http://127.5.118.130:45019/ using document root <none> and password file <none>
I20260620 14:53:40.515856 20609 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:40.515899 20609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:40.517308 20609 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:40.517879 20624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:40.518090 20609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:40.518182 20609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "ea490d5a70a94ba89549a37c83fff72a"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "f345e10e929e568e4d3d4e3667710fc0"
server_key_iv: "e8d8409166edd4837cf22a82198e85b9"
server_key_version: "encryptionkey@0"
I20260620 14:53:40.518532 20609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:40.536515 20609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:40.538749 20609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:40.538877 20609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:40.539194 20609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:40.539572 20609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:40.539616 20609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:40.539639 20609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:40.539687 20609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:40.545917 20609 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:33551
I20260620 14:53:40.545962 20737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:33551 every 8 connection(s)
I20260620 14:53:40.546344 20609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:40.547291 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 20609
I20260620 14:53:40.547578 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:40805
--local_ip_for_outbound_sockets=127.5.118.131
--tserver_master_addrs=127.5.118.190:38567
--webserver_port=34963
--webserver_interface=127.5.118.131
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:40.547574 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.547118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:50367)
0620 14:53:40.547297 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.547387 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
W20260620 14:53:40.548069 20738 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:38567 (0 consecutive failures): Network error: Failed to ping master at 127.5.118.190:38567: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
I20260620 14:53:40.548379 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.548277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:52693)
0620 14:53:40.548326 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.548352 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:53:40.548646 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.548560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:48203)
0620 14:53:40.548604 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.548627 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:53:40.548746 20738 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:53:40.629944 20741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:40.630178 20741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:40.630204 20741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:40.631340 20741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:40.631376 20741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:40.631402 20741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:53:40.632947 20741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:40805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=34963
--tserver_master_addrs=127.5.118.190:38567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:40.633147 20741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:40.633327 20741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:40.638404 20746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:40.638404 20749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:40.638487 20747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:40.638624 20741 server_base.cc:1034] running on GCE node
I20260620 14:53:40.638837 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:40.639041 20741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:40.640252 20741 hybrid_clock.cc:648] HybridClock initialized: now 1781967220640240 us; error 46 us; skew 500 ppm
Jun 20 14:53:40 dist-test-slave-2m5f krb5kdc[19302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:40.642771 20741 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:53:40.643249 20741 webserver.cc:460] Webserver started at http://127.5.118.131:34963/ using document root <none> and password file <none>
I20260620 14:53:40.643436 20741 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:40.643476 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:40.644870 20741 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:53:40.645524 20756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:40.645714 20741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:53:40.645803 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "4106f645342b444fb6ed90c80b02bb18"
format_stamp: "Formatted at 2026-06-20 14:53:39 on dist-test-slave-2m5f"
server_key: "8441e4645b34b62fb9af2419e5706158"
server_key_iv: "983e0ed7073a2423f29208b1633845e7"
server_key_version: "encryptionkey@0"
I20260620 14:53:40.646186 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:40.653102 20741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:40.654924 20741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:40.655048 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:40.655223 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:40.655550 20741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:40.655586 20741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:40.655609 20741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:40.655658 20741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:40.661857 20741 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:40805
I20260620 14:53:40.661892 20869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:40805 every 8 connection(s)
I20260620 14:53:40.662293 20741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:40.662760 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 20741
I20260620 14:53:40.663900 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.663157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:53511)
0620 14:53:40.663409 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.663527 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
W20260620 14:53:40.664162 20870 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:38567 (0 consecutive failures): Network error: Failed to ping master at 127.5.118.190:38567: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
I20260620 14:53:40.664472 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.664388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:53957)
0620 14:53:40.664421 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.664454 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 14:53:40.664752 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:40.664658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:42843)
0620 14:53:40.664705 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:40.664731 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:53:40.664868 20870 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:53:41.505693 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:41.505424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:50609)
0620 14:53:41.505586 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:41.505631 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
I20260620 14:53:41.549418 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:41.549180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:36555)
0620 14:53:41.549317 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:41.549362 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260620 14:53:41.665412 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:41.665206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:48933)
0620 14:53:41.665327 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:41.665363 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260620 14:53:42.138991 20877 raft_consensus.cc:482] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:42.139163 20877 raft_consensus.cc:504] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:42.139593 20877 leader_election.cc:288] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805), ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551)
Jun 20 14:53:42 dist-test-slave-2m5f krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.131@KRBTEST.COM
Jun 20 14:53:42 dist-test-slave-2m5f krb5kdc[19302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.130@KRBTEST.COM
I20260620 14:53:42.152904 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:42.139978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:43531 (local address 127.5.118.130:33551)
0620 14:53:42.140126 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:42.140138 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:42.140158 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:53:42.140724 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:42.140726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:42.140769 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:42.140924 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:42.142036 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.143495 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.145077 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.145719 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.145844 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:42.149130 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:42.149134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:42.149134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:42.149146 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:42.151083 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:42.151561 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:42.151564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:42.151564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:42.151663 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:42.151906 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:42.151907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:42.151908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:42.152239 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:42.152350 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:42.152641 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 14:53:42.152782 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":51,"threads_started":1}
I20260620 14:53:42.152904 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:42.139755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:40805 (local address 127.5.118.129:55037)
0620 14:53:42.139871 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:42.139882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:42.139966 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:42.141027 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:42.141030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:42.141558 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:42.142086 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:42.142089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.143624 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.143625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:42.144971 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:42.144974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.145833 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.145834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:42.145984 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:42.146665 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:42.146676 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:42.148663 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:42.150933 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:42.150941 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:42.150944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:42.151457 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:42.151716 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:42.151717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:42.151718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:42.151841 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:42.152330 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:42.152337 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:42.152472 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 14:53:42.152589 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:53:42.152796 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":51,"thread_start_us":51,"threads_started":1}
I20260620 14:53:42.152930 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:42.139847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:55037 (local address 127.5.118.131:40805)
0620 14:53:42.140005 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:42.140020 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:42.140058 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:53:42.140626 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:42.140629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:42.140728 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:42.140875 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:42.142198 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.143497 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.145078 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.145719 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.145845 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:42.148785 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:42.148791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:42.148791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:42.148805 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:42.150839 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:42.151538 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:42.151541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:42.151541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:42.151639 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:42.151935 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:42.151936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:42.151936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:42.152252 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:42.152339 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:42.152650 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 14:53:42.152791 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260620 14:53:42.153029 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:42.139770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:33551 (local address 127.5.118.129:43531)
0620 14:53:42.140019 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:42.140030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:42.140092 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:42.141027 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:42.141030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:42.141558 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:42.141881 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:42.141891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.143633 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.143633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:42.144971 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:42.144974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:42.145834 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:42.145834 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:42.145984 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:42.146684 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:42.146688 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:42.149025 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:42.151180 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:42.151181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:42.151181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:42.151457 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:42.151744 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:42.151745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:42.151746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:42.151839 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:42.152335 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:42.152341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:42.152472 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 14:53:42.152589 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:53:42.152884 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":213,"thread_start_us":64,"threads_started":1}
W20260620 14:53:42.153792 20493 leader_election.cc:341] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805): Not found: Tablet not found: a06d14febdbe49cb945178d6270ba7cb
W20260620 14:53:42.154212 20492 leader_election.cc:341] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551): Not found: Tablet not found: a06d14febdbe49cb945178d6270ba7cb
I20260620 14:53:42.154277 20492 leader_election.cc:302] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b7957b2eff4649b72bf7295887eeb2; no voters: 4106f645342b444fb6ed90c80b02bb18, ea490d5a70a94ba89549a37c83fff72a
I20260620 14:53:42.154371 20877 raft_consensus.cc:2726] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:53:42.506486 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:42.506279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:55141)
0620 14:53:42.506359 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:42.506413 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:53:42.550191 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:42.549940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:60619)
0620 14:53:42.550090 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:42.550143 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260620 14:53:42.666114 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:42.665898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:39469)
0620 14:53:42.666032 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:42.666073 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260620 14:53:43.507308 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:43.507089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:54185)
0620 14:53:43.507223 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:43.507264 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260620 14:53:43.550913 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:43.550712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:37291)
0620 14:53:43.550833 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:43.550870 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260620 14:53:43.666831 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:43.666611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:37355)
0620 14:53:43.666743 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:43.666784 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260620 14:53:44.263998 20887 raft_consensus.cc:482] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:44.264142 20887 raft_consensus.cc:504] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:44.264401 20887 leader_election.cc:288] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805), ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551)
W20260620 14:53:44.264883 20493 leader_election.cc:341] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805): Not found: Tablet not found: a06d14febdbe49cb945178d6270ba7cb
W20260620 14:53:44.264961 20492 leader_election.cc:341] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551): Not found: Tablet not found: a06d14febdbe49cb945178d6270ba7cb
I20260620 14:53:44.264977 20492 leader_election.cc:302] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b7957b2eff4649b72bf7295887eeb2; no voters: 4106f645342b444fb6ed90c80b02bb18, ea490d5a70a94ba89549a37c83fff72a
I20260620 14:53:44.265077 20887 raft_consensus.cc:2726] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:53:44.508083 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:44.507822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:41149)
0620 14:53:44.507977 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:44.508022 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260620 14:53:44.551682 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:44.551432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:45785)
0620 14:53:44.551589 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:44.551628 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260620 14:53:44.667605 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:44.667338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:58099)
0620 14:53:44.667502 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:44.667544 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260620 14:53:45.508946 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:45.508696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:50171)
0620 14:53:45.508843 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:45.508894 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
I20260620 14:53:45.552546 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:45.552246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:51461)
0620 14:53:45.552455 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:45.552498 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
I20260620 14:53:45.668375 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:45.668140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:58299)
0620 14:53:45.668286 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:45.668328 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260620 14:53:46.509799 20894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:46.509548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:46207)
0620 14:53:46.509705 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:46.509751 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260620 14:53:46.553411 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:46.553087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:58199)
0620 14:53:46.553326 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:46.553362 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":178,"threads_started":1}
I20260620 14:53:46.669153 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:46.668893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:43183)
0620 14:53:46.669059 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:46.669105 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260620 14:53:46.682075 20897 raft_consensus.cc:482] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:46.682206 20897 raft_consensus.cc:504] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:46.682428 20897 leader_election.cc:288] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805), ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551)
W20260620 14:53:46.682855 20492 leader_election.cc:341] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551): Not found: Tablet not found: a06d14febdbe49cb945178d6270ba7cb
W20260620 14:53:46.682962 20493 leader_election.cc:341] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805): Not found: Tablet not found: a06d14febdbe49cb945178d6270ba7cb
I20260620 14:53:46.682991 20493 leader_election.cc:302] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b7957b2eff4649b72bf7295887eeb2; no voters: 4106f645342b444fb6ed90c80b02bb18, ea490d5a70a94ba89549a37c83fff72a
I20260620 14:53:46.683090 20897 raft_consensus.cc:2726] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:53:47.510667 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:47.510404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:54307)
0620 14:53:47.510575 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:47.510623 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260620 14:53:47.554204 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:47.553960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:32781)
0620 14:53:47.554112 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:47.554156 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260620 14:53:47.670051 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:47.669812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:52133)
0620 14:53:47.669956 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:47.670003 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260620 14:53:48.511569 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:48.511311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:51097)
0620 14:53:48.511469 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:48.511521 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260620 14:53:48.555241 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:48.554868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:33765)
0620 14:53:48.555127 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:48.555180 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260620 14:53:48.670900 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:48.670627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:40693)
0620 14:53:48.670802 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:48.670849 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260620 14:53:49.512596 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:49.512350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:45431)
0620 14:53:49.512491 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:49.512544 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260620 14:53:49.556269 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:49.555939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:41065)
0620 14:53:49.556113 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:49.556193 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":69,"threads_started":1}
I20260620 14:53:49.671706 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:49.671446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:59849)
0620 14:53:49.671611 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:49.671660 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":78,"threads_started":1}
I20260620 14:53:50.513458 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:50.513200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:57063)
0620 14:53:50.513365 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:50.513413 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
I20260620 14:53:50.557057 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:50.556770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:47733)
0620 14:53:50.556958 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:50.557008 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260620 14:53:50.672760 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:50.672492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:60109)
0620 14:53:50.672656 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:50.672710 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260620 14:53:51.475275 20910 raft_consensus.cc:482] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:51.475402 20910 raft_consensus.cc:504] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4106f645342b444fb6ed90c80b02bb18" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40805 } } peers { permanent_uuid: "ea490d5a70a94ba89549a37c83fff72a" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33551 } } peers { permanent_uuid: "61b7957b2eff4649b72bf7295887eeb2" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 38877 } }
I20260620 14:53:51.475580 20910 leader_election.cc:288] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805), ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551)
W20260620 14:53:51.476161 20493 leader_election.cc:341] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4106f645342b444fb6ed90c80b02bb18 (127.5.118.131:40805): Not found: Tablet not found: a06d14febdbe49cb945178d6270ba7cb
W20260620 14:53:51.476258 20492 leader_election.cc:341] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ea490d5a70a94ba89549a37c83fff72a (127.5.118.130:33551): Not found: Tablet not found: a06d14febdbe49cb945178d6270ba7cb
I20260620 14:53:51.476287 20493 leader_election.cc:302] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 61b7957b2eff4649b72bf7295887eeb2; no voters: 4106f645342b444fb6ed90c80b02bb18, ea490d5a70a94ba89549a37c83fff72a
I20260620 14:53:51.476390 20910 raft_consensus.cc:2726] T a06d14febdbe49cb945178d6270ba7cb P 61b7957b2eff4649b72bf7295887eeb2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:53:51.514340 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:51.514098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:35559)
0620 14:53:51.514241 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:51.514291 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260620 14:53:51.557982 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:51.557690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:50637)
0620 14:53:51.557857 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:51.557926 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":72,"threads_started":1}
I20260620 14:53:51.673781 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:51.673502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:39441)
0620 14:53:51.673680 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:51.673731 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260620 14:53:52.515199 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:52.514970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:37151)
0620 14:53:52.515107 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:52.515153 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260620 14:53:52.558805 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:52.558558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:52373)
0620 14:53:52.558712 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:52.558757 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260620 14:53:52.674611 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:52.674341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:49353)
0620 14:53:52.674517 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:52.674563 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
I20260620 14:53:53.516206 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:53.515874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:40313)
0620 14:53:53.516075 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:53.516122 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260620 14:53:53.559630 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:53.559401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:54159)
0620 14:53:53.559540 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:53.559581 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260620 14:53:53.675491 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:53.675242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:33965)
0620 14:53:53.675385 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:53.675442 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260620 14:53:54.517051 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:54.516825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:39545)
0620 14:53:54.516960 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:54.517006 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260620 14:53:54.560449 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:54.560165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:45345)
0620 14:53:54.560355 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:54.560402 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":120,"threads_started":1}
I20260620 14:53:54.676491 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:54.676140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.131:34583)
0620 14:53:54.676380 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:54.676441 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":143,"threads_started":1}
I20260620 14:53:55.517990 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:55.517746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.129:34359)
0620 14:53:55.517899 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:55.517943 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
I20260620 14:53:55.561223 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:55.561005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38567 (local address 127.5.118.130:47985)
0620 14:53:55.561142 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:55.561178 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:38567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260620 14:53:55.666978 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 20445
I20260620 14:53:55.672312 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 20609
I20260620 14:53:55.677098 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 20741
2026-06-20T14:53:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18172 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:55 dist-test-slave-2m5f krb5kdc[20927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:55 dist-test-slave-2m5f krb5kdc[20927](info): set up 2 sockets
Jun 20 14:53:55 dist-test-slave-2m5f krb5kdc[20927](info): commencing operation
krb5kdc: starting...
W20260620 14:53:56.748387 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:53:56 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:53:56.870848 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42153
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:42153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:56.956280 20940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:56.956516 20940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:56.956543 20940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:56.957861 20940 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:53:56.957908 20940 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:53:56.957921 20940 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:53:56.957933 20940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:56.957952 20940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:56.957966 20940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:56.957978 20940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:56.957990 20940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:56.959538 20940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:42153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:56.959800 20940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:56.960047 20940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:56.965229 20945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:56.965222 20948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:56.965354 20940 server_base.cc:1034] running on GCE node
W20260620 14:53:56.965229 20946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:56.965950 20940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:56.966367 20940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:56.967546 20940 hybrid_clock.cc:648] HybridClock initialized: now 1781967236967543 us; error 46 us; skew 500 ppm
Jun 20 14:53:56 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967236, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:56.970336 20940 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:53:56.970808 20940 webserver.cc:460] Webserver started at http://127.5.118.190:38987/ using document root <none> and password file <none>
I20260620 14:53:56.971051 20940 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:56.971118 20940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:56.971261 20940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:56.972532 20940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "fe4b10df2e2a46e582882bdfd04e1a0a"
format_stamp: "Formatted at 2026-06-20 14:53:56 on dist-test-slave-2m5f"
server_key: "6c52e3a5ecfdf0e8218c4cc52c55268f"
server_key_iv: "320bc602e674cbd47514085b7c47e7db"
server_key_version: "encryptionkey@0"
I20260620 14:53:56.972905 20940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "fe4b10df2e2a46e582882bdfd04e1a0a"
format_stamp: "Formatted at 2026-06-20 14:53:56 on dist-test-slave-2m5f"
server_key: "6c52e3a5ecfdf0e8218c4cc52c55268f"
server_key_iv: "320bc602e674cbd47514085b7c47e7db"
server_key_version: "encryptionkey@0"
I20260620 14:53:56.974409 20940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:53:56.975378 20955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:56.975615 20940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:56.975749 20940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "fe4b10df2e2a46e582882bdfd04e1a0a"
format_stamp: "Formatted at 2026-06-20 14:53:56 on dist-test-slave-2m5f"
server_key: "6c52e3a5ecfdf0e8218c4cc52c55268f"
server_key_iv: "320bc602e674cbd47514085b7c47e7db"
server_key_version: "encryptionkey@0"
I20260620 14:53:56.975831 20940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:56.994686 20940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:56.997538 20940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:56.997694 20940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:57.001825 20940 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:42153
I20260620 14:53:57.001892 21007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:42153 every 8 connection(s)
I20260620 14:53:57.002249 20940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:57.003207 21008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:57.005853 21008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Bootstrap starting.
I20260620 14:53:57.006532 21008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:57.006538 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 20940
I20260620 14:53:57.006639 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:53:57.006886 5594 external_mini_cluster.cc:1428] Setting key 4678c98fc6d7dac20ba666ef067f0ca5
I20260620 14:53:57.006906 21008 log.cc:824] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:57.007803 21008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: No bootstrap required, opened a new log
I20260620 14:53:57.010651 21008 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:53:57.010835 21008 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:57.010867 21008 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4b10df2e2a46e582882bdfd04e1a0a, State: Initialized, Role: FOLLOWER
I20260620 14:53:57.011009 21008 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:53:57.011114 21008 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:57.011161 21008 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:57.011217 21008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:57.011996 21008 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:53:57.012121 21008 leader_election.cc:302] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe4b10df2e2a46e582882bdfd04e1a0a; no voters:
I20260620 14:53:57.012356 21008 leader_election.cc:288] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:57.012414 21013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:57.012689 21013 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 1 LEADER]: Becoming Leader. State: Replica: fe4b10df2e2a46e582882bdfd04e1a0a, State: Running, Role: LEADER
I20260620 14:53:57.012800 21013 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:53:57.012663 21008 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:57.013316 21014 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } } }
I20260620 14:53:57.013424 21014 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:57.013706 21017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:57.014250 21017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:57.013279 21015 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe4b10df2e2a46e582882bdfd04e1a0a. Latest consensus state: current_term: 1 leader_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } } }
I20260620 14:53:57.015228 21015 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: This master's current role is: LEADER
Jun 20 14:53:57 dist-test-slave-2m5f krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:57.016305 21017 catalog_manager.cc:1347] Generated new cluster ID: 77379055d75f4814aeba65c61cb5e8f6
I20260620 14:53:57.016361 21017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:57.020309 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.007435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.5.118.190:42153)
0620 14:53:57.007631 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.007642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.007676 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:53:57.008331 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.008342 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.008414 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.008618 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:57.009288 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.010953 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.012218 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.012965 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.013116 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.016635 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:57.016640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:57.016640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:57.016653 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:57.018551 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.018967 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.018970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.018970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.019071 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.019304 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.019305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.019305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.019563 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:57.019737 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.019854 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.020028 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260620 14:53:57.022846 21017 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:57.023413 21017 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:57.029569 21017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Generated new TSK 0
I20260620 14:53:57.029774 21017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:53:57.059309 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:57.146148 21034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:57.146389 21034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:57.146427 21034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:57.147600 21034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:57.147667 21034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:57.147710 21034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:53:57.149220 21034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:57.149492 21034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:57.149730 21034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:57.154819 21040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:57.154953 21034 server_base.cc:1034] running on GCE node
W20260620 14:53:57.155058 21042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:57.154819 21039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:57.155328 21034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:57.155560 21034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:57.156700 21034 hybrid_clock.cc:648] HybridClock initialized: now 1781967237156679 us; error 38 us; skew 500 ppm
Jun 20 14:53:57 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:57.159467 21034 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:53:57.159943 21034 webserver.cc:460] Webserver started at http://127.5.118.129:46689/ using document root <none> and password file <none>
I20260620 14:53:57.160262 21034 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:57.160331 21034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:57.160483 21034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:57.161641 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "74e2fe2faf24468aa46c9d234c1312f9"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "0c7ea80c8d92a9c7a7417f899b036b69"
server_key_iv: "867a961d514f8b4fca2fd65235148ca9"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.162021 21034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "74e2fe2faf24468aa46c9d234c1312f9"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "0c7ea80c8d92a9c7a7417f899b036b69"
server_key_iv: "867a961d514f8b4fca2fd65235148ca9"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.163426 21034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:53:57.164250 21049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:57.164414 21034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:53:57.164515 21034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "74e2fe2faf24468aa46c9d234c1312f9"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "0c7ea80c8d92a9c7a7417f899b036b69"
server_key_iv: "867a961d514f8b4fca2fd65235148ca9"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.164594 21034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:57.188073 21034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:57.190270 21034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:57.190424 21034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:57.190725 21034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:57.191062 21034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:57.191116 21034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:57.191151 21034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:57.191182 21034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:57.197160 21034 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:35807
I20260620 14:53:57.197310 21162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:35807 every 8 connection(s)
I20260620 14:53:57.197576 21034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:53:57 dist-test-slave-2m5f krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:57.205219 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 21034
I20260620 14:53:57.205309 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:53:57.205523 5594 external_mini_cluster.cc:1428] Setting key 26548226a7b883ed8d6b55a3b1294143
I20260620 14:53:57.207762 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.198483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:53783 (local address 127.5.118.190:42153)
0620 14:53:57.198607 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.198610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.198796 (+ 186us) server_negotiation.cc:407] Connection header received
0620 14:53:57.199362 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.199363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.199375 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.199433 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:57.200486 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.201009 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.202051 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.202262 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.202338 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.204666 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:57.204667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:57.204667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:57.204675 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:57.206303 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.206727 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.206729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.206729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.206788 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.207040 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.207041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.207041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.207161 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:57.207243 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.207605 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.207693 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:53:57.208026 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.198452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:53783)
0620 14:53:57.198652 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:57.198705 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:57.199250 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:57.199520 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:57.199524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:57.200114 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:57.200388 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.200393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.201121 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.201122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.201949 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.201951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.202359 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.202360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.202498 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.203058 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:57.203067 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:57.204566 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:57.206402 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:57.206409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:57.206411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:57.206651 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:57.206859 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:57.206860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:57.206860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:57.206970 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:57.207232 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:57.207239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:57.207414 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 14:53:57.207561 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 14:53:57.207756 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
I20260620 14:53:57.208539 21163 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42153
I20260620 14:53:57.208640 21163 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:57.208840 21163 heartbeater.cc:511] Master 127.5.118.190:42153 requested a full tablet report, sending...
I20260620 14:53:57.209388 20972 ts_manager.cc:194] Registered new tserver with Master: 74e2fe2faf24468aa46c9d234c1312f9 (127.5.118.129:35807)
I20260620 14:53:57.210085 20972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:53783
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:53:57.231560 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:57.316888 21168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:57.317107 21168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:57.317133 21168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:57.318256 21168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:57.318288 21168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:57.318313 21168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:53:57.319769 21168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:57.319960 21168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:57.320211 21168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:57.325284 21174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:57.325299 21176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:57.325306 21173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:57.325331 21168 server_base.cc:1034] running on GCE node
I20260620 14:53:57.325664 21168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:57.325919 21168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:57.327121 21168 hybrid_clock.cc:648] HybridClock initialized: now 1781967237327087 us; error 44 us; skew 500 ppm
Jun 20 14:53:57 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:57.329907 21168 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:53:57.330377 21168 webserver.cc:460] Webserver started at http://127.5.118.130:38875/ using document root <none> and password file <none>
I20260620 14:53:57.330623 21168 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:57.330673 21168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:57.330801 21168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:57.331935 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "feb35f8c90244891a6bb1893ebf3a925"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "ee64bcc7a597a4a98494249cf3032c88"
server_key_iv: "9c64f2defad8ce19a8710d0b7b283a98"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.332324 21168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "feb35f8c90244891a6bb1893ebf3a925"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "ee64bcc7a597a4a98494249cf3032c88"
server_key_iv: "9c64f2defad8ce19a8710d0b7b283a98"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.333757 21168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:53:57.334599 21183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:57.334774 21168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:53:57.334863 21168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "feb35f8c90244891a6bb1893ebf3a925"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "ee64bcc7a597a4a98494249cf3032c88"
server_key_iv: "9c64f2defad8ce19a8710d0b7b283a98"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.334923 21168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:57.346796 21168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:57.349202 21168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:57.349327 21168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:57.349519 21168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:57.349856 21168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:57.349892 21168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:57.349915 21168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:57.349959 21168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:57.355871 21168 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:40119
I20260620 14:53:57.355952 21296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:40119 every 8 connection(s)
I20260620 14:53:57.356329 21168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:53:57.356806 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 21168
I20260620 14:53:57.356936 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:53:57.357206 5594 external_mini_cluster.cc:1428] Setting key c44e96ed8fbd8e83aebe0eb6d92906a2
Jun 20 14:53:57 dist-test-slave-2m5f krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
I20260620 14:53:57.368285 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.357151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:56829 (local address 127.5.118.190:42153)
0620 14:53:57.357223 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.357226 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.357552 (+ 326us) server_negotiation.cc:407] Connection header received
0620 14:53:57.358272 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.358273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.358282 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.358310 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:57.359703 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.360544 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.362226 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.362461 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.362526 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.365183 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:57.365185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:57.365185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:57.365192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:57.366729 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.367133 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.367134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.367134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.367190 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.367547 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.367548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.367548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.367677 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:57.367757 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.368059 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.368148 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:53:57.368568 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.357132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:56829)
0620 14:53:57.357356 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:57.357409 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:57.358152 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:57.358404 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:57.358408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:57.359276 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:57.359589 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.359597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.360700 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.360701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.362096 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.362100 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.362576 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.362577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.362758 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.363357 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:57.363369 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:57.365040 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:57.366826 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:57.366840 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:57.366842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:57.367061 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:57.367304 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:57.367305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:57.367305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:57.367456 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:57.367747 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:57.367754 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:57.367905 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 14:53:57.368029 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 14:53:57.368263 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
I20260620 14:53:57.369124 21297 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42153
I20260620 14:53:57.369247 21297 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:57.369455 21297 heartbeater.cc:511] Master 127.5.118.190:42153 requested a full tablet report, sending...
I20260620 14:53:57.369910 20972 ts_manager.cc:194] Registered new tserver with Master: feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119)
I20260620 14:53:57.370311 20972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:56829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:53:57.383275 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:57.467118 21302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:57.467378 21302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:57.467420 21302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:57.468706 21302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:57.468779 21302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:57.468824 21302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:53:57.470314 21302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:53:57.470592 21302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:57.470821 21302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:57.475895 21308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:57.475929 21307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:57.475895 21310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:57.476208 21302 server_base.cc:1034] running on GCE node
I20260620 14:53:57.476420 21302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:57.476630 21302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:57.477797 21302 hybrid_clock.cc:648] HybridClock initialized: now 1781967237477787 us; error 36 us; skew 500 ppm
Jun 20 14:53:57 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:57.480458 21302 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:53:57.480937 21302 webserver.cc:460] Webserver started at http://127.5.118.131:41141/ using document root <none> and password file <none>
I20260620 14:53:57.481168 21302 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:57.481231 21302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:57.481383 21302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:57.482478 21302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "b5b46f6ba9cb4316941f10e912543721"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "c5dfbd8317af0a7c21992a2428755bb0"
server_key_iv: "80a527c584cdab4a20496cd4df32bc02"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.482836 21302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "b5b46f6ba9cb4316941f10e912543721"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "c5dfbd8317af0a7c21992a2428755bb0"
server_key_iv: "80a527c584cdab4a20496cd4df32bc02"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.484133 21302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:53:57.484916 21317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:57.485082 21302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:53:57.485178 21302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "b5b46f6ba9cb4316941f10e912543721"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "c5dfbd8317af0a7c21992a2428755bb0"
server_key_iv: "80a527c584cdab4a20496cd4df32bc02"
server_key_version: "encryptionkey@0"
I20260620 14:53:57.485255 21302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:57.495112 21302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:57.497206 21302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:57.497367 21302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:57.497604 21302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:53:57.497910 21302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:53:57.497983 21302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:57.498049 21302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:53:57.498081 21302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:57.504242 21302 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:38903
I20260620 14:53:57.504293 21430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:38903 every 8 connection(s)
I20260620 14:53:57.504608 21302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:53:57.508292 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 21302
I20260620 14:53:57.508417 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:53:57.508634 5594 external_mini_cluster.cc:1428] Setting key eff597a93d8520560bb3000e025f719a
Jun 20 14:53:57 dist-test-slave-2m5f krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:53:57.514923 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.505418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:48463 (local address 127.5.118.190:42153)
0620 14:53:57.505481 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.505483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.505691 (+ 208us) server_negotiation.cc:407] Connection header received
0620 14:53:57.506295 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.506297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.506309 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.506370 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:57.507352 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.507915 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.509176 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.509472 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.509536 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.511711 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:57.511713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:57.511713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:57.511721 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:57.513294 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.513745 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.513747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.513747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.513817 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.514228 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.514229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.514229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.514351 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:57.514413 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.514746 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.514827 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:53:57.515331 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.505402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:48463)
0620 14:53:57.505567 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:57.505603 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:57.506182 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:57.506457 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:57.506461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:57.507049 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:57.507249 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.507253 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.508035 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.508036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.509112 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.509115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.509556 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.509557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.509691 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.510210 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:57.510218 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:57.511617 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:57.513384 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:57.513392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:57.513395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:57.513651 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:57.513946 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:57.513947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:57.513947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:57.514077 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:57.514445 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:57.514451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:57.514587 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 14:53:57.514727 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 14:53:57.515007 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260620 14:53:57.515913 21431 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42153
I20260620 14:53:57.516005 21431 heartbeater.cc:464] Registering TS with master...
I20260620 14:53:57.516233 21431 heartbeater.cc:511] Master 127.5.118.190:42153 requested a full tablet report, sending...
I20260620 14:53:57.516688 20972 ts_manager.cc:194] Registered new tserver with Master: b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903)
I20260620 14:53:57.517063 20972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:48463
I20260620 14:53:57.520557 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:53:57.528522 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.521935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.5.118.190:42153)
0620 14:53:57.521977 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.521980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.522140 (+ 160us) server_negotiation.cc:407] Connection header received
0620 14:53:57.522165 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.522165 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.522175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.522231 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:57.522748 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.523243 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.524020 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.524500 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.524624 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.525890 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:57.525891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:57.525892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:57.525898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:57.527299 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.527638 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.527639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.527640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.527698 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:57.527909 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:57.527910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:57.527910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:57.528073 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:57.528154 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.528378 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.528452 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:53:57.530367 20972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:53:57.532416 20972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:53:57.539009 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.534686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:38903 (local address 127.0.0.1:50514)
0620 14:53:57.535115 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:57.535141 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:57.535233 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:57.535449 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:57.535452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:57.535454 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:57.535536 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.535538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.536610 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.536610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.537790 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.537792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.538686 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.538687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.538779 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.538796 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:53:57.538875 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:53:57.538931 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":85,"threads_started":1}
I20260620 14:53:57.539033 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.534893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.5.118.131:38903)
0620 14:53:57.535061 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.535067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.535209 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:53:57.535327 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.535330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.535345 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.535433 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:57.535646 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.536508 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.537875 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.538600 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.538687 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.538815 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.538890 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.538931 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260620 14:53:57.539187 21443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.534684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:35807 (local address 127.0.0.1:43254)
0620 14:53:57.535095 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:57.535142 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:57.535213 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:57.535462 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:57.535463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:57.535466 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:57.535575 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.535578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.537016 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.537017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.538059 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.538061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.538962 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.538963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.539032 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.539038 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:53:57.539069 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 14:53:57.539110 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":31,"thread_start_us":67,"threads_started":1}
I20260620 14:53:57.539361 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.534915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:40119 (local address 127.0.0.1:41380)
0620 14:53:57.535313 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:57.535320 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:57.535366 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:57.535766 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:57.535767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:57.535768 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:57.535807 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.535808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.537078 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.537079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.538234 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.538237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.539150 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.539151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.539205 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.539209 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:53:57.539258 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 14:53:57.539299 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":81,"threads_started":1}
I20260620 14:53:57.539572 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.535023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.5.118.130:40119)
0620 14:53:57.535188 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.535193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.535539 (+ 346us) server_negotiation.cc:407] Connection header received
0620 14:53:57.535590 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.535593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.535610 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.535682 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:57.535887 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.536986 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.538321 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.538993 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.539054 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.539119 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.539422 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.539474 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260620 14:53:57.539870 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.534814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.5.118.129:35807)
0620 14:53:57.535251 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.535255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.535280 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:53:57.535323 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.535325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.535342 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.535663 (+ 321us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:57.535750 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.536904 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.538144 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.538858 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.539628 (+ 770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.539695 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.539722 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.539771 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":86,"threads_started":1}
I20260620 14:53:57.539991 21365 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5d55a60e1f4176a1d05f02ffa1176e (DEFAULT_TABLE table=test-table [id=3d5cfb4bb3074c94b3c72fa2e878fb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:57.540241 21231 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5d55a60e1f4176a1d05f02ffa1176e (DEFAULT_TABLE table=test-table [id=3d5cfb4bb3074c94b3c72fa2e878fb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:57.540416 21365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5d55a60e1f4176a1d05f02ffa1176e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:57.540455 21097 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5d55a60e1f4176a1d05f02ffa1176e (DEFAULT_TABLE table=test-table [id=3d5cfb4bb3074c94b3c72fa2e878fb51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:53:57.540537 21231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5d55a60e1f4176a1d05f02ffa1176e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:57.540720 21097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5d55a60e1f4176a1d05f02ffa1176e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:57.543270 21448 tablet_bootstrap.cc:492] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Bootstrap starting.
I20260620 14:53:57.543823 21450 tablet_bootstrap.cc:492] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721: Bootstrap starting.
I20260620 14:53:57.543973 21448 tablet_bootstrap.cc:654] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:57.544553 21450 tablet_bootstrap.cc:654] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:57.544620 21448 log.cc:824] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:57.545042 21449 tablet_bootstrap.cc:492] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925: Bootstrap starting.
I20260620 14:53:57.545176 21450 log.cc:824] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:57.545545 21448 tablet_bootstrap.cc:492] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: No bootstrap required, opened a new log
I20260620 14:53:57.545636 21449 tablet_bootstrap.cc:654] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:57.545629 21448 ts_tablet_manager.cc:1397] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:57.545981 21450 tablet_bootstrap.cc:492] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721: No bootstrap required, opened a new log
I20260620 14:53:57.546048 21450 ts_tablet_manager.cc:1397] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:53:57.546363 21449 log.cc:824] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:57.547029 21449 tablet_bootstrap.cc:492] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925: No bootstrap required, opened a new log
I20260620 14:53:57.547111 21449 ts_tablet_manager.cc:1397] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:53:57.547709 21448 raft_consensus.cc:348] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } }
I20260620 14:53:57.547910 21448 raft_consensus.cc:374] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:57.547966 21448 raft_consensus.cc:729] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e2fe2faf24468aa46c9d234c1312f9, State: Initialized, Role: FOLLOWER
I20260620 14:53:57.547973 21450 raft_consensus.cc:348] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } }
I20260620 14:53:57.548101 21448 consensus_queue.cc:260] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } }
I20260620 14:53:57.548148 21450 raft_consensus.cc:374] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:57.548214 21450 raft_consensus.cc:729] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b46f6ba9cb4316941f10e912543721, State: Initialized, Role: FOLLOWER
I20260620 14:53:57.548370 21450 consensus_queue.cc:260] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } }
I20260620 14:53:57.548437 21163 heartbeater.cc:503] Master 127.5.118.190:42153 was elected leader, sending a full tablet report...
I20260620 14:53:57.548673 21431 heartbeater.cc:503] Master 127.5.118.190:42153 was elected leader, sending a full tablet report...
I20260620 14:53:57.548439 21448 ts_tablet_manager.cc:1428] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:57.548980 21450 ts_tablet_manager.cc:1428] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 14:53:57.548946 21449 raft_consensus.cc:348] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } }
I20260620 14:53:57.549289 21449 raft_consensus.cc:374] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:57.549342 21449 raft_consensus.cc:729] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb35f8c90244891a6bb1893ebf3a925, State: Initialized, Role: FOLLOWER
I20260620 14:53:57.549456 21449 consensus_queue.cc:260] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } }
I20260620 14:53:57.549758 21297 heartbeater.cc:503] Master 127.5.118.190:42153 was elected leader, sending a full tablet report...
I20260620 14:53:57.549741 21449 ts_tablet_manager.cc:1428] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:57.581739 21455 raft_consensus.cc:482] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:53:57.581878 21455 raft_consensus.cc:504] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } }
I20260620 14:53:57.582227 21455 leader_election.cc:288] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74e2fe2faf24468aa46c9d234c1312f9 (127.5.118.129:35807), feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119)
I20260620 14:53:57.585291 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.582510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:45833 (local address 127.5.118.130:40119)
0620 14:53:57.582553 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.582555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.582645 (+ 90us) server_negotiation.cc:407] Connection header received
0620 14:53:57.582693 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.582694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.582706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.582768 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:57.582934 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.583405 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.584441 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.584846 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.584904 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.584912 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.585170 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.585213 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:53:57.585299 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.582381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:40119 (local address 127.5.118.131:45833)
0620 14:53:57.582577 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:57.582587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:57.582640 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:57.582796 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:57.582797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:57.582799 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:57.582870 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.582872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.583485 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.583486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.584360 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.584363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.585024 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.585025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.585101 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.585105 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:53:57.585160 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 14:53:57.585193 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260620 14:53:57.585408 21444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.582437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:34577 (local address 127.5.118.129:35807)
0620 14:53:57.582483 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.582485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.582495 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:53:57.582527 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.582527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.582538 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.582620 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:53:57.582809 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.583341 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.584510 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.585017 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.585099 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.585107 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.585269 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.585312 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:53:57.585758 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.582360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:35807 (local address 127.5.118.131:34577)
0620 14:53:57.582405 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:57.582418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:57.582469 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:57.582648 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:57.582649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:57.582651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:53:57.582732 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.582734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.583425 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.583426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.584409 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:57.584413 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.585102 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.585103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:57.585194 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.585197 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:53:57.585253 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:53:57.585292 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":40}
I20260620 14:53:57.585712 21251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5d55a60e1f4176a1d05f02ffa1176e" candidate_uuid: "b5b46f6ba9cb4316941f10e912543721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb35f8c90244891a6bb1893ebf3a925" is_pre_election: true
I20260620 14:53:57.585888 21251 raft_consensus.cc:2454] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b46f6ba9cb4316941f10e912543721 in term 0.
I20260620 14:53:57.586107 21320 leader_election.cc:302] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5b46f6ba9cb4316941f10e912543721, feb35f8c90244891a6bb1893ebf3a925; no voters:
I20260620 14:53:57.586120 21117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5d55a60e1f4176a1d05f02ffa1176e" candidate_uuid: "b5b46f6ba9cb4316941f10e912543721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e2fe2faf24468aa46c9d234c1312f9" is_pre_election: true
I20260620 14:53:57.586236 21455 raft_consensus.cc:2781] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:53:57.586263 21117 raft_consensus.cc:2454] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b46f6ba9cb4316941f10e912543721 in term 0.
I20260620 14:53:57.586295 21455 raft_consensus.cc:482] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:53:57.586313 21455 raft_consensus.cc:3037] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:57.587031 21455 raft_consensus.cc:504] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } }
I20260620 14:53:57.587183 21455 leader_election.cc:288] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [CANDIDATE]: Term 1 election: Requested vote from peers 74e2fe2faf24468aa46c9d234c1312f9 (127.5.118.129:35807), feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119)
I20260620 14:53:57.587396 21117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5d55a60e1f4176a1d05f02ffa1176e" candidate_uuid: "b5b46f6ba9cb4316941f10e912543721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e2fe2faf24468aa46c9d234c1312f9"
I20260620 14:53:57.587450 21251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5d55a60e1f4176a1d05f02ffa1176e" candidate_uuid: "b5b46f6ba9cb4316941f10e912543721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb35f8c90244891a6bb1893ebf3a925"
I20260620 14:53:57.587507 21117 raft_consensus.cc:3037] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:57.587520 21251 raft_consensus.cc:3037] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:57.588500 21117 raft_consensus.cc:2454] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b46f6ba9cb4316941f10e912543721 in term 1.
I20260620 14:53:57.588500 21251 raft_consensus.cc:2454] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b46f6ba9cb4316941f10e912543721 in term 1.
I20260620 14:53:57.588685 21321 leader_election.cc:302] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74e2fe2faf24468aa46c9d234c1312f9, b5b46f6ba9cb4316941f10e912543721; no voters:
I20260620 14:53:57.588814 21455 raft_consensus.cc:2781] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:57.588989 21455 raft_consensus.cc:686] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 1 LEADER]: Becoming Leader. State: Replica: b5b46f6ba9cb4316941f10e912543721, State: Running, Role: LEADER
I20260620 14:53:57.589183 21455 consensus_queue.cc:237] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } }
I20260620 14:53:57.590104 20971 catalog_manager.cc:5495] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5b46f6ba9cb4316941f10e912543721 (127.5.118.131). New cstate: current_term: 1 leader_uuid: "b5b46f6ba9cb4316941f10e912543721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } health_report { overall_health: UNKNOWN } } }
I20260620 14:53:57.596745 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.594268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.5.118.131:38903)
0620 14:53:57.594314 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:57.594315 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:57.594324 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:53:57.594398 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:57.594399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:57.594413 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:57.594469 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:57.594626 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.595143 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.595912 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:57.596147 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:57.596290 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:57.596412 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:57.596509 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:57.596570 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:57.596603 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 14:53:57.596621 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:53:57.598541 21117 raft_consensus.cc:1261] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Refusing update from remote peer b5b46f6ba9cb4316941f10e912543721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:57.598544 21251 raft_consensus.cc:1261] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925 [term 1 FOLLOWER]: Refusing update from remote peer b5b46f6ba9cb4316941f10e912543721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:57.598902 21458 consensus_queue.cc:1035] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:57.599037 21455 consensus_queue.cc:1035] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:57.601735 21461 mvcc.cc:205] Tried to move back new op lower bound from 7298937805200977920 to 7298937805166063616. Current Snapshot: MvccSnapshot[applied={T|T < 7298937805200977920}]
I20260620 14:53:57.602329 21463 mvcc.cc:205] Tried to move back new op lower bound from 7298937805200977920 to 7298937805166063616. Current Snapshot: MvccSnapshot[applied={T|T < 7298937805200977920}]
I20260620 14:53:57.603978 20971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:57.604205 20971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:53:57.604987 20971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Sending DeleteTablet for 3 replicas of tablet 0c5d55a60e1f4176a1d05f02ffa1176e
I20260620 14:53:57.605293 21365 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5d55a60e1f4176a1d05f02ffa1176e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:57 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:50514
I20260620 14:53:57.605289 21097 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5d55a60e1f4176a1d05f02ffa1176e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:57 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:43254
I20260620 14:53:57.605400 21231 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5d55a60e1f4176a1d05f02ffa1176e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:57 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:41380
I20260620 14:53:57.605412 21230 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5d55a60e1f4176a1d05f02ffa1176e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:57 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:41380
I20260620 14:53:57.605633 21473 tablet_replica.cc:330] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925: stopping tablet replica
I20260620 14:53:57.605674 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 21034
I20260620 14:53:57.605736 21473 raft_consensus.cc:2229] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:53:57.606441 21472 tablet_replica.cc:330] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721: stopping tablet replica
I20260620 14:53:57.606630 21472 raft_consensus.cc:2229] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:53:57.606794 21472 raft_consensus.cc:2258] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:57.608841 21473 raft_consensus.cc:2258] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:53:57.609256 21472 ts_tablet_manager.cc:1905] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:57.609333 21473 ts_tablet_manager.cc:1905] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:53:57.611167 21473 ts_tablet_manager.cc:1918] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:53:57.611253 21473 log.cc:1196] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/0c5d55a60e1f4176a1d05f02ffa1176e
I20260620 14:53:57.611569 21473 ts_tablet_manager.cc:1939] T 0c5d55a60e1f4176a1d05f02ffa1176e P feb35f8c90244891a6bb1893ebf3a925: Deleting consensus metadata
I20260620 14:53:57.611708 21472 ts_tablet_manager.cc:1918] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:53:57.611832 21472 log.cc:1196] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/0c5d55a60e1f4176a1d05f02ffa1176e
I20260620 14:53:57.612066 20958 catalog_manager.cc:4841] TS feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119): tablet 0c5d55a60e1f4176a1d05f02ffa1176e (table test-table [id=3d5cfb4bb3074c94b3c72fa2e878fb51]) successfully deleted
W20260620 14:53:57.612108 20959 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:35807 (error 108)
I20260620 14:53:57.612272 21472 ts_tablet_manager.cc:1939] T 0c5d55a60e1f4176a1d05f02ffa1176e P b5b46f6ba9cb4316941f10e912543721: Deleting consensus metadata
W20260620 14:53:57.612443 20958 catalog_manager.cc:4808] TS feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119): delete failed for tablet 0c5d55a60e1f4176a1d05f02ffa1176e because the tablet was not found. No further retry: Not found: Tablet not found: 0c5d55a60e1f4176a1d05f02ffa1176e
I20260620 14:53:57.612840 20958 catalog_manager.cc:4841] TS b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903): tablet 0c5d55a60e1f4176a1d05f02ffa1176e (table test-table [id=3d5cfb4bb3074c94b3c72fa2e878fb51]) successfully deleted
I20260620 14:53:57.612869 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 21168
I20260620 14:53:57.613526 21442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:57.612965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:35807 (local address 127.0.0.1:43258)
0620 14:53:57.613443 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:57.613484 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457}
W20260620 14:53:57.613829 20959 catalog_manager.cc:4584] TS 74e2fe2faf24468aa46c9d234c1312f9 (127.5.118.129:35807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c5d55a60e1f4176a1d05f02ffa1176e: Network error: Client connection negotiation failed: client connection to 127.5.118.129:35807: connect: Connection refused (error 111)
I20260620 14:53:57.619372 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 21302
I20260620 14:53:57.624469 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 20940
I20260620 14:54:17.630415 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42153
--webserver_interface=127.5.118.190
--webserver_port=38987
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:42153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:17.713281 21475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:17.713517 21475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:17.713541 21475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:17.714821 21475 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:54:17.714871 21475 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:54:17.714885 21475 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:54:17.714897 21475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:17.714910 21475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:17.714923 21475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:17.714936 21475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:17.714999 21475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:17.716688 21475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:42153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=38987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:17.716984 21475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:17.717250 21475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:17.727173 21482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:17.727161 21481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:17.727185 21475 server_base.cc:1034] running on GCE node
W20260620 14:54:17.727161 21484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:17.727777 21475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:17.728130 21475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:17.729354 21475 hybrid_clock.cc:648] HybridClock initialized: now 1781967257729330 us; error 40 us; skew 500 ppm
Jun 20 14:54:17 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967257, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:17.738361 21475 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:54:17.738957 21475 webserver.cc:460] Webserver started at http://127.5.118.190:38987/ using document root <none> and password file <none>
I20260620 14:54:17.739267 21475 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:17.739362 21475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:17.741053 21475 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:17.741829 21491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:17.742038 21475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:17.742184 21475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "fe4b10df2e2a46e582882bdfd04e1a0a"
format_stamp: "Formatted at 2026-06-20 14:53:56 on dist-test-slave-2m5f"
server_key: "6c52e3a5ecfdf0e8218c4cc52c55268f"
server_key_iv: "320bc602e674cbd47514085b7c47e7db"
server_key_version: "encryptionkey@0"
I20260620 14:54:17.742624 21475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:17.753062 21475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:17.755712 21475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:17.755900 21475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:17.760422 21475 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:42153
I20260620 14:54:17.760548 21543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:42153 every 8 connection(s)
I20260620 14:54:17.760921 21475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:17.762586 21544 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:54:17.763324 21544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Bootstrap starting.
I20260620 14:54:17.765981 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 21475
I20260620 14:54:17.766292 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:35807
--local_ip_for_outbound_sockets=127.5.118.129
--tserver_master_addrs=127.5.118.190:42153
--webserver_port=46689
--webserver_interface=127.5.118.129
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:54:17.772403 21544 log.cc:824] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:17.774416 21544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:54:17.774772 21544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Bootstrap complete.
I20260620 14:54:17.777251 21544 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:54:17.777524 21544 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4b10df2e2a46e582882bdfd04e1a0a, State: Initialized, Role: FOLLOWER
I20260620 14:54:17.777726 21544 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:54:17.777808 21544 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:17.777868 21544 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:17.777920 21544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:54:17.778977 21544 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:54:17.779107 21544 leader_election.cc:302] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe4b10df2e2a46e582882bdfd04e1a0a; no voters:
I20260620 14:54:17.779325 21544 leader_election.cc:288] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:54:17.779362 21548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:54:17.779477 21548 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 2 LEADER]: Becoming Leader. State: Replica: fe4b10df2e2a46e582882bdfd04e1a0a, State: Running, Role: LEADER
I20260620 14:54:17.779620 21544 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:17.779639 21548 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:54:17.779992 21550 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe4b10df2e2a46e582882bdfd04e1a0a. Latest consensus state: current_term: 2 leader_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } } }
I20260620 14:54:17.779999 21549 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } } }
I20260620 14:54:17.780200 21550 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:17.780202 21549 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:17.780913 21557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:17.781742 21557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:17.781888 21557 catalog_manager.cc:1259] Loaded cluster ID: 77379055d75f4814aeba65c61cb5e8f6
I20260620 14:54:17.781948 21557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:17.782810 21557 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:17.783015 21557 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Loaded TSK: 0
I20260620 14:54:17.790828 21557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Generated new TSK 1
I20260620 14:54:17.790957 21557 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:54:17.868834 21546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:17.869081 21546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:17.869125 21546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:17.870362 21546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:17.870435 21546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:17.870483 21546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:54:17.872153 21546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:35807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=46689
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:17.872507 21546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:17.872772 21546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:17.879231 21571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:17.879318 21572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:17.879251 21574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:17.879637 21546 server_base.cc:1034] running on GCE node
I20260620 14:54:17.879863 21546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:17.880229 21546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:17.881409 21546 hybrid_clock.cc:648] HybridClock initialized: now 1781967257881393 us; error 38 us; skew 500 ppm
Jun 20 14:54:17 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967257, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:17.884083 21546 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:54:17.884629 21546 webserver.cc:460] Webserver started at http://127.5.118.129:46689/ using document root <none> and password file <none>
I20260620 14:54:17.884904 21546 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:17.884963 21546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:17.886391 21546 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:54:17.887102 21581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:17.887276 21546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:54:17.887384 21546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "74e2fe2faf24468aa46c9d234c1312f9"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "0c7ea80c8d92a9c7a7417f899b036b69"
server_key_iv: "867a961d514f8b4fca2fd65235148ca9"
server_key_version: "encryptionkey@0"
I20260620 14:54:17.887792 21546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:17.894524 21546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:17.896919 21546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:17.897068 21546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:17.897351 21546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:17.897876 21588 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:54:17.899334 21546 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:54:17.899385 21546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:54:17.899421 21546 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:54:17.900087 21546 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:54:17.900132 21546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:54:17.900205 21588 tablet_bootstrap.cc:492] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Bootstrap starting.
I20260620 14:54:17.907970 21546 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:35807
I20260620 14:54:17.908267 21695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:35807 every 8 connection(s)
I20260620 14:54:17.908563 21546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:17.911736 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 21546
I20260620 14:54:17.912035 21588 log.cc:824] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:17.912101 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:40119
--local_ip_for_outbound_sockets=127.5.118.130
--tserver_master_addrs=127.5.118.190:42153
--webserver_port=38875
--webserver_interface=127.5.118.130
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:54:17 dist-test-slave-2m5f krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967257, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:54:17.920655 21588 tablet_bootstrap.cc:492] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:54:17.921089 21588 tablet_bootstrap.cc:492] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Bootstrap complete.
I20260620 14:54:17.922506 21588 ts_tablet_manager.cc:1397] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.011s
I20260620 14:54:17.922784 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.909337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:51461)
0620 14:54:17.909661 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:17.909725 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:17.910396 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:17.911188 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:17.911194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:17.911617 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:17.912005 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.912012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.913349 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.913350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.914291 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.914294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.914989 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.914991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.915123 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.915762 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:17.915771 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:17.917346 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:17.920025 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:17.920033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:17.920036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:17.920510 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:17.920982 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:17.920983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:17.920983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:17.921196 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:17.921862 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:17.921869 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:17.922046 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 14:54:17.922146 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:54:17.922362 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":104,"threads_started":1}
I20260620 14:54:17.924355 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.909483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:51461 (local address 127.5.118.190:42153)
0620 14:54:17.910657 (+ 1174us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:17.910667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:17.910722 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:54:17.910872 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:17.910879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:17.910941 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:17.911099 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:17.912166 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.913173 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.914408 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.914870 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.915001 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.917486 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:17.917491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:17.917492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:17.917507 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:17.919899 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.920626 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.920630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.920631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.920887 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.921366 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.921367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.921367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.921762 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:17.923786 (+ 2024us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:17.923881 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:54:17.924054 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1113,"thread_start_us":109,"threads_started":1}
I20260620 14:54:17.924253 21588 raft_consensus.cc:348] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } }
I20260620 14:54:17.924623 21588 raft_consensus.cc:729] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e2fe2faf24468aa46c9d234c1312f9, State: Initialized, Role: FOLLOWER
I20260620 14:54:17.924813 21588 consensus_queue.cc:260] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } }
I20260620 14:54:17.925140 21588 ts_tablet_manager.cc:1428] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260620 14:54:17.925354 21696 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42153
I20260620 14:54:17.925452 21696 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:17.925643 21696 heartbeater.cc:511] Master 127.5.118.190:42153 requested a full tablet report, sending...
I20260620 14:54:17.926586 21508 ts_manager.cc:194] Registered new tserver with Master: 74e2fe2faf24468aa46c9d234c1312f9 (127.5.118.129:35807)
I20260620 14:54:17.928009 21508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:51461
I20260620 14:54:17.929064 21696 heartbeater.cc:503] Master 127.5.118.190:42153 was elected leader, sending a full tablet report...
Jun 20 14:54:17 dist-test-slave-2m5f krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967257, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.129@KRBTEST.COM
I20260620 14:54:17.939338 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.927315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:35807 (local address 127.0.0.1:43544)
0620 14:54:17.928284 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:17.928327 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:17.928400 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:17.928709 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:17.928714 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:17.929080 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:17.929258 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.929261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.930686 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.930687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.931782 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:17.931786 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.932527 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.932528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:17.932624 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.933240 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:17.933246 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:17.935176 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:17.937858 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:17.937867 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:17.937869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:17.938180 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:17.938450 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:17.938451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:17.938451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:17.938529 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:17.938881 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:17.938890 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:17.938966 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 14:54:17.939076 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:54:17.939222 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"thread_start_us":111,"threads_started":1}
I20260620 14:54:17.939816 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:17.927598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.5.118.129:35807)
0620 14:54:17.928535 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:17.928540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:17.928550 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:54:17.928586 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:17.928588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:17.928602 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:17.929349 (+ 747us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:17.929402 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.930587 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.931891 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:17.932420 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:17.932504 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:17.935303 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:17.935314 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:17.935315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:17.935324 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:17.937736 (+ 2412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.938287 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.938291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.938292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.938367 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:17.938600 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:17.938601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:17.938601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:17.938801 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:17.938890 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:17.939122 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 14:54:17.939275 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":90,"threads_started":1}
I20260620 14:54:17.940376 21630 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5d55a60e1f4176a1d05f02ffa1176e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:53:57 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:43544
I20260620 14:54:17.941090 21705 tablet_replica.cc:330] stopping tablet replica
I20260620 14:54:17.941217 21705 raft_consensus.cc:2229] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:17.941288 21705 raft_consensus.cc:2258] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:17.941640 21705 ts_tablet_manager.cc:1905] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:17.944124 21705 ts_tablet_manager.cc:1918] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:17.944245 21705 log.cc:1196] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal/wals/0c5d55a60e1f4176a1d05f02ffa1176e
I20260620 14:54:17.944478 21705 ts_tablet_manager.cc:1939] T 0c5d55a60e1f4176a1d05f02ffa1176e P 74e2fe2faf24468aa46c9d234c1312f9: Deleting consensus metadata
I20260620 14:54:17.945086 21495 catalog_manager.cc:4841] TS 74e2fe2faf24468aa46c9d234c1312f9 (127.5.118.129:35807): tablet 0c5d55a60e1f4176a1d05f02ffa1176e (table test-table [id=3d5cfb4bb3074c94b3c72fa2e878fb51]) successfully deleted
W20260620 14:54:18.008071 21701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:18.008323 21701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:18.008348 21701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:18.009461 21701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:18.009496 21701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:18.009522 21701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:54:18.010936 21701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:40119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=38875
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:18.011128 21701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:18.011307 21701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:18.016723 21711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:18.016717 21710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:18.016759 21701 server_base.cc:1034] running on GCE node
W20260620 14:54:18.016717 21713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:18.017262 21701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:18.017486 21701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:18.018704 21701 hybrid_clock.cc:648] HybridClock initialized: now 1781967258018678 us; error 41 us; skew 500 ppm
Jun 20 14:54:18 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:18.021684 21701 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:54:18.022291 21701 webserver.cc:460] Webserver started at http://127.5.118.130:38875/ using document root <none> and password file <none>
I20260620 14:54:18.022552 21701 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:18.022619 21701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:18.024255 21701 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:54:18.024940 21720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.025163 21701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:18.025279 21701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "feb35f8c90244891a6bb1893ebf3a925"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "ee64bcc7a597a4a98494249cf3032c88"
server_key_iv: "9c64f2defad8ce19a8710d0b7b283a98"
server_key_version: "encryptionkey@0"
I20260620 14:54:18.025646 21701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:18.039674 21701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:18.042276 21701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:18.042456 21701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:18.042713 21701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:18.043057 21701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:18.043115 21701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.043161 21701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:18.043184 21701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.049829 21701 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:40119
I20260620 14:54:18.049867 21833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:40119 every 8 connection(s)
I20260620 14:54:18.050243 21701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:18.051142 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 21701
I20260620 14:54:18.051517 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:38903
--local_ip_for_outbound_sockets=127.5.118.131
--tserver_master_addrs=127.5.118.190:42153
--webserver_port=41141
--webserver_interface=127.5.118.131
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:54:18 dist-test-slave-2m5f krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:54:18.062034 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.050981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:46909 (local address 127.5.118.190:42153)
0620 14:54:18.051037 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.051039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.051394 (+ 355us) server_negotiation.cc:407] Connection header received
0620 14:54:18.051897 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.051899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.051933 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.052012 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.053509 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.054055 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.055700 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.055911 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.055977 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.058754 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.058756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.058757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.058766 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.060464 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.060958 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.060959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.060960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.061023 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.061299 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.061300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.061300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.061424 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.061522 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.061839 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.061932 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:54:18.062254 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.050994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:46909)
0620 14:54:18.051244 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.051291 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.051785 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.052136 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.052141 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.053048 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:18.053375 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.053383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.054163 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.054165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.055599 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.055602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.056043 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.056044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.056247 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.056822 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:18.056834 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:18.058637 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:18.060601 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.060609 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.060611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.060859 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.061096 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.061097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.061097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.061218 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.061493 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:18.061500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:18.061675 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.061807 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.061984 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":104,"threads_started":1}
I20260620 14:54:18.062799 21834 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42153
I20260620 14:54:18.062918 21834 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:18.063114 21834 heartbeater.cc:511] Master 127.5.118.190:42153 requested a full tablet report, sending...
I20260620 14:54:18.063565 21508 ts_manager.cc:194] Registered new tserver with Master: feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119)
I20260620 14:54:18.063946 21508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:46909
W20260620 14:54:18.141093 21837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:18.141327 21837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:18.141356 21837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:18.142675 21837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:18.142726 21837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:18.142758 21837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:54:18.144444 21837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:38903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=41141
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:18.144726 21837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:18.145030 21837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:18.150609 21843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:18.150595 21845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:18.150626 21837 server_base.cc:1034] running on GCE node
W20260620 14:54:18.150595 21842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:18.151046 21837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:18.151283 21837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:18.152424 21837 hybrid_clock.cc:648] HybridClock initialized: now 1781967258152412 us; error 31 us; skew 500 ppm
Jun 20 14:54:18 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:18.155038 21837 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:54:18.155512 21837 webserver.cc:460] Webserver started at http://127.5.118.131:41141/ using document root <none> and password file <none>
I20260620 14:54:18.155781 21837 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:18.155872 21837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:18.157351 21837 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:18.157982 21852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.158183 21837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:18.158288 21837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "b5b46f6ba9cb4316941f10e912543721"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "c5dfbd8317af0a7c21992a2428755bb0"
server_key_iv: "80a527c584cdab4a20496cd4df32bc02"
server_key_version: "encryptionkey@0"
I20260620 14:54:18.158658 21837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:18.170466 21837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:18.172951 21837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:18.173141 21837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:18.173404 21837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:18.173745 21837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:18.173802 21837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.173843 21837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:18.173888 21837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:18.180465 21837 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:38903
I20260620 14:54:18.180512 21965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:38903 every 8 connection(s)
I20260620 14:54:18.180895 21837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:54:18 dist-test-slave-2m5f krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967258, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:54:18.187395 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 21837
I20260620 14:54:18.191064 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.181781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:38155 (local address 127.5.118.190:42153)
0620 14:54:18.181862 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.181864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.182062 (+ 198us) server_negotiation.cc:407] Connection header received
0620 14:54:18.182539 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.182540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.182551 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.182611 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.183549 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.184041 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.185148 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.185339 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.185395 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.187598 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.187599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.187600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.187608 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.189407 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.189834 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.189836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.189836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.189908 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.190225 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.190226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.190227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.190352 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.190410 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.190853 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.190951 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:54:18.191268 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.181746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:38155)
0620 14:54:18.181923 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.181967 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.182449 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.182673 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.182676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.183250 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:18.183451 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.183456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.184131 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.184132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.185059 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.185062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.185415 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.185415 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.185519 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.186065 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:18.186074 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:18.187493 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:18.189492 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.189498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.189500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.189739 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.189997 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:18.189998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:18.189998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:18.190143 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:18.190425 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:18.190431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:18.190607 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.190816 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.190996 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260620 14:54:18.192075 21966 heartbeater.cc:347] Connected to a master server at 127.5.118.190:42153
I20260620 14:54:18.192216 21966 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:18.192431 21966 heartbeater.cc:511] Master 127.5.118.190:42153 requested a full tablet report, sending...
I20260620 14:54:18.192883 21508 ts_manager.cc:194] Registered new tserver with Master: b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903)
I20260620 14:54:18.193403 21508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:38155
I20260620 14:54:18.195465 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.187857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.5.118.190:42153)
0620 14:54:18.187989 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.187991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.188084 (+ 93us) server_negotiation.cc:407] Connection header received
0620 14:54:18.188165 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.188166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.188206 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.188277 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.188996 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.189870 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.190995 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.191493 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.191554 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.193046 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.193047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.193047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.193053 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.194495 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.194879 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.194879 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.194880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.194927 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.195077 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.195077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.195078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.195163 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.195272 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.195320 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.195379 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":88,"threads_started":1}
I20260620 14:54:18.196396 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:54:18.199388 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.196693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.5.118.190:42153)
0620 14:54:18.196728 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.196730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.196850 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:54:18.196925 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.196926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.196962 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.197046 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:18.197158 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.197632 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.198423 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.198714 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.198797 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.198968 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:18.199019 (+ 51us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 14:54:18.199311 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53616: token expired
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 14:54:18.199445 21970 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53616: token expired
W20260620 14:54:18.199585 21434 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.118.190:42153)
I20260620 14:54:18.206380 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.199888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.5.118.190:42153)
0620 14:54:18.199921 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.199922 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.199931 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:54:18.199973 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.199974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.199983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.200030 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:18.200594 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.201184 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.202003 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.202204 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.202275 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.203510 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:18.203512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:18.203512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:18.203518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:18.205190 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.205560 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.205562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.205562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.205622 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:18.205833 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:18.205834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:18.205835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:18.205957 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:18.206067 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.206181 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.206278 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:54:18.207713 21508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:18.209450 21508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:18.211768 21630 tablet_service.cc:1467] Processing CreateTablet for tablet 51e74398063541b1861918f7fd461124 (DEFAULT_TABLE table=test-table [id=14b50665d9f84b3697e553f758d1471a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:18.211968 21630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e74398063541b1861918f7fd461124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:18.213517 21588 tablet_bootstrap.cc:492] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: Bootstrap starting.
I20260620 14:54:18.213901 21588 tablet_bootstrap.cc:654] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:18.214440 21588 tablet_bootstrap.cc:492] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: No bootstrap required, opened a new log
I20260620 14:54:18.214504 21588 ts_tablet_manager.cc:1397] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:18.214891 21588 raft_consensus.cc:348] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:18.214985 21588 raft_consensus.cc:374] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:18.215045 21588 raft_consensus.cc:729] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e2fe2faf24468aa46c9d234c1312f9, State: Initialized, Role: FOLLOWER
I20260620 14:54:18.215101 21588 consensus_queue.cc:260] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:18.215197 21588 ts_tablet_manager.cc:1428] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:18.216154 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.211630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:40119 (local address 127.0.0.1:50290)
0620 14:54:18.211906 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.211978 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.212025 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.212307 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.212308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.212311 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:18.212361 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.212362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.213441 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.213442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.214812 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.214815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.215881 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.215882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.215942 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.215966 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.216034 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.216077 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":135,"threads_started":1}
I20260620 14:54:18.216264 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.211732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.5.118.130:40119)
0620 14:54:18.211895 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.211900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.212078 (+ 178us) server_negotiation.cc:407] Connection header received
0620 14:54:18.212124 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.212127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.212150 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.212296 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:18.212437 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.213354 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.214899 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.215817 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.215883 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.215943 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.216076 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.216119 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260620 14:54:18.217123 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.211636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.5.118.131:38903)
0620 14:54:18.213580 (+ 1944us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.213585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.213596 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:54:18.213634 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.213636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.213651 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.213763 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:18.213905 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.214913 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.216054 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.216756 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.216814 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.216849 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.217037 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.217067 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1909,"thread_start_us":117,"threads_started":1}
I20260620 14:54:18.217134 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.211536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:38903 (local address 127.0.0.1:38694)
0620 14:54:18.211811 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.211824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.211867 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.213788 (+ 1921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.213789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.213791 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:18.213838 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.213840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.214985 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.214986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.215961 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.215963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.216836 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.216837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.216954 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.216958 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.217030 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.217073 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260620 14:54:18.216991 21768 tablet_service.cc:1467] Processing CreateTablet for tablet 51e74398063541b1861918f7fd461124 (DEFAULT_TABLE table=test-table [id=14b50665d9f84b3697e553f758d1471a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:18.217399 21768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e74398063541b1861918f7fd461124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:18.220373 21976 tablet_bootstrap.cc:492] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925: Bootstrap starting.
I20260620 14:54:18.221064 21976 tablet_bootstrap.cc:654] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:18.221197 21900 tablet_service.cc:1467] Processing CreateTablet for tablet 51e74398063541b1861918f7fd461124 (DEFAULT_TABLE table=test-table [id=14b50665d9f84b3697e553f758d1471a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:18.221561 21900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51e74398063541b1861918f7fd461124. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:18.221686 21976 log.cc:824] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:18.222513 21976 tablet_bootstrap.cc:492] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925: No bootstrap required, opened a new log
I20260620 14:54:18.222573 21976 ts_tablet_manager.cc:1397] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:54:18.224592 21976 raft_consensus.cc:348] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:18.224781 21976 raft_consensus.cc:374] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:18.224828 21976 raft_consensus.cc:729] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb35f8c90244891a6bb1893ebf3a925, State: Initialized, Role: FOLLOWER
I20260620 14:54:18.224977 21976 consensus_queue.cc:260] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:18.225251 21978 tablet_bootstrap.cc:492] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721: Bootstrap starting.
I20260620 14:54:18.225301 21976 ts_tablet_manager.cc:1428] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:18.225428 21834 heartbeater.cc:503] Master 127.5.118.190:42153 was elected leader, sending a full tablet report...
I20260620 14:54:18.226039 21978 tablet_bootstrap.cc:654] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:18.226699 21978 log.cc:824] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:18.227397 21978 tablet_bootstrap.cc:492] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721: No bootstrap required, opened a new log
I20260620 14:54:18.227453 21978 ts_tablet_manager.cc:1397] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:54:18.229103 21978 raft_consensus.cc:348] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:18.229313 21978 raft_consensus.cc:374] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:18.229365 21978 raft_consensus.cc:729] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b46f6ba9cb4316941f10e912543721, State: Initialized, Role: FOLLOWER
I20260620 14:54:18.229540 21978 consensus_queue.cc:260] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:18.229892 21978 ts_tablet_manager.cc:1428] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:54:18.229897 21966 heartbeater.cc:503] Master 127.5.118.190:42153 was elected leader, sending a full tablet report...
I20260620 14:54:18.240890 21981 raft_consensus.cc:482] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:18.241050 21981 raft_consensus.cc:504] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:18.241446 21981 leader_election.cc:288] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119), 74e2fe2faf24468aa46c9d234c1312f9 (127.5.118.129:35807)
I20260620 14:54:18.244426 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.241819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:37117 (local address 127.5.118.129:35807)
0620 14:54:18.241867 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.241869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.241878 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:54:18.241905 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.241906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.241934 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.242001 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:18.242163 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.242648 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.243593 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.244018 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.244123 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.244236 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.244302 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.244344 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:54:18.244465 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.241631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:35807 (local address 127.5.118.131:37117)
0620 14:54:18.241690 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.241705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.241762 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.242005 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.242007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.242008 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:18.242086 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.242088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.242734 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.242735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.243494 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.243496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.244122 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.244123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.244230 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.244234 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.244298 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.244343 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":19}
I20260620 14:54:18.244454 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.241796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:55755 (local address 127.5.118.130:40119)
0620 14:54:18.241849 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.241851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.241940 (+ 89us) server_negotiation.cc:407] Connection header received
0620 14:54:18.242016 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.242017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.242026 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.242076 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:18.242268 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.242743 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.243678 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.244066 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.244152 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.244161 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.244342 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.244382 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 14:54:18.244482 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.241637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:40119 (local address 127.5.118.131:55755)
0620 14:54:18.241852 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.241862 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:18.241948 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:18.242077 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:18.242078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:18.242079 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:18.242163 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.242164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.242830 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.242831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.243579 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:18.243580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.244151 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.244152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:18.244259 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.244264 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:54:18.244337 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:54:18.244376 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":14,"thread_start_us":136,"threads_started":1}
I20260620 14:54:18.244804 21788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e74398063541b1861918f7fd461124" candidate_uuid: "b5b46f6ba9cb4316941f10e912543721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb35f8c90244891a6bb1893ebf3a925" is_pre_election: true
I20260620 14:54:18.244798 21650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e74398063541b1861918f7fd461124" candidate_uuid: "b5b46f6ba9cb4316941f10e912543721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e2fe2faf24468aa46c9d234c1312f9" is_pre_election: true
I20260620 14:54:18.244964 21650 raft_consensus.cc:2454] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b46f6ba9cb4316941f10e912543721 in term 0.
I20260620 14:54:18.244962 21788 raft_consensus.cc:2454] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5b46f6ba9cb4316941f10e912543721 in term 0.
I20260620 14:54:18.245188 21855 leader_election.cc:302] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5b46f6ba9cb4316941f10e912543721, feb35f8c90244891a6bb1893ebf3a925; no voters:
I20260620 14:54:18.245357 21981 raft_consensus.cc:2781] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:18.245431 21981 raft_consensus.cc:482] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:18.245467 21981 raft_consensus.cc:3037] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:18.246295 21981 raft_consensus.cc:504] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:18.246454 21981 leader_election.cc:288] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [CANDIDATE]: Term 1 election: Requested vote from peers feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119), 74e2fe2faf24468aa46c9d234c1312f9 (127.5.118.129:35807)
I20260620 14:54:18.246652 21788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e74398063541b1861918f7fd461124" candidate_uuid: "b5b46f6ba9cb4316941f10e912543721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb35f8c90244891a6bb1893ebf3a925"
I20260620 14:54:18.246646 21650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51e74398063541b1861918f7fd461124" candidate_uuid: "b5b46f6ba9cb4316941f10e912543721" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74e2fe2faf24468aa46c9d234c1312f9"
I20260620 14:54:18.246757 21788 raft_consensus.cc:3037] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:18.246773 21650 raft_consensus.cc:3037] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:18.247486 21788 raft_consensus.cc:2454] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b46f6ba9cb4316941f10e912543721 in term 1.
I20260620 14:54:18.247541 21650 raft_consensus.cc:2454] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5b46f6ba9cb4316941f10e912543721 in term 1.
I20260620 14:54:18.247690 21855 leader_election.cc:302] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5b46f6ba9cb4316941f10e912543721, feb35f8c90244891a6bb1893ebf3a925; no voters:
I20260620 14:54:18.247788 21981 raft_consensus.cc:2781] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:18.247980 21981 raft_consensus.cc:686] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 1 LEADER]: Becoming Leader. State: Replica: b5b46f6ba9cb4316941f10e912543721, State: Running, Role: LEADER
I20260620 14:54:18.248099 21981 consensus_queue.cc:237] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:18.248973 21507 catalog_manager.cc:5495] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5b46f6ba9cb4316941f10e912543721 (127.5.118.131). New cstate: current_term: 1 leader_uuid: "b5b46f6ba9cb4316941f10e912543721" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } health_report { overall_health: UNKNOWN } } }
I20260620 14:54:18.263409 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.260686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.5.118.131:38903)
0620 14:54:18.260735 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:18.260737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:18.260748 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:54:18.260823 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:18.260823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:18.260843 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:18.260922 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:18.261046 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.261620 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.262552 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:18.262753 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:18.262857 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:18.263028 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:18.263142 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:18.263245 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:18.263262 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 14:54:18.263286 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:54:18.265424 21788 raft_consensus.cc:1261] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925 [term 1 FOLLOWER]: Refusing update from remote peer b5b46f6ba9cb4316941f10e912543721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:18.265439 21650 raft_consensus.cc:1261] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Refusing update from remote peer b5b46f6ba9cb4316941f10e912543721: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:18.265884 21983 consensus_queue.cc:1035] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:18.266032 21981 consensus_queue.cc:1035] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:18.268464 21985 mvcc.cc:205] Tried to move back new op lower bound from 7298937889852559360 to 7298937889785094144. Current Snapshot: MvccSnapshot[applied={T|T < 7298937889852559360}]
I20260620 14:54:18.269105 21987 mvcc.cc:205] Tried to move back new op lower bound from 7298937889852559360 to 7298937889785094144. Current Snapshot: MvccSnapshot[applied={T|T < 7298937889852559360 or (T in {7298937889852559360})}]
I20260620 14:54:18.270953 21507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:18.271063 21507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:18.271873 21507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Sending DeleteTablet for 3 replicas of tablet 51e74398063541b1861918f7fd461124
I20260620 14:54:18.272320 21768 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e74398063541b1861918f7fd461124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:18 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:50290
I20260620 14:54:18.272315 21900 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e74398063541b1861918f7fd461124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:18 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:38694
I20260620 14:54:18.272605 21630 tablet_service.cc:1514] Processing DeleteTablet for tablet 51e74398063541b1861918f7fd461124 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:18 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:43544
I20260620 14:54:18.272647 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 21546
I20260620 14:54:18.272718 21705 tablet_replica.cc:330] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: stopping tablet replica
I20260620 14:54:18.272810 21705 raft_consensus.cc:2229] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:18.272881 21705 raft_consensus.cc:2258] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:18.273411 21995 tablet_replica.cc:330] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925: stopping tablet replica
I20260620 14:54:18.273514 21995 raft_consensus.cc:2229] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:18.273614 21995 raft_consensus.cc:2258] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:18.272763 21994 tablet_replica.cc:330] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721: stopping tablet replica
I20260620 14:54:18.274085 21995 ts_tablet_manager.cc:1905] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:18.275851 21995 ts_tablet_manager.cc:1918] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:18.275947 21995 log.cc:1196] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/51e74398063541b1861918f7fd461124
I20260620 14:54:18.276260 21995 ts_tablet_manager.cc:1939] T 51e74398063541b1861918f7fd461124 P feb35f8c90244891a6bb1893ebf3a925: Deleting consensus metadata
I20260620 14:54:18.276687 21494 catalog_manager.cc:4841] TS feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119): tablet 51e74398063541b1861918f7fd461124 (table test-table [id=14b50665d9f84b3697e553f758d1471a]) successfully deleted
I20260620 14:54:18.276882 21994 raft_consensus.cc:2229] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:18.277060 21994 raft_consensus.cc:2258] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:18.277482 21994 ts_tablet_manager.cc:1905] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:54:18.277904 21495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:35807 (error 108)
I20260620 14:54:18.279034 21994 ts_tablet_manager.cc:1918] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:18.279126 21994 log.cc:1196] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/51e74398063541b1861918f7fd461124
I20260620 14:54:18.279189 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 21701
I20260620 14:54:18.279539 21703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:18.279367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:35807 (local address 127.0.0.1:43548)
0620 14:54:18.279446 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:18.279502 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:35807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 14:54:18.279986 21994 ts_tablet_manager.cc:1939] T 51e74398063541b1861918f7fd461124 P b5b46f6ba9cb4316941f10e912543721: Deleting consensus metadata
W20260620 14:54:18.279824 21495 catalog_manager.cc:4584] TS 74e2fe2faf24468aa46c9d234c1312f9 (127.5.118.129:35807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51e74398063541b1861918f7fd461124: Network error: Client connection negotiation failed: client connection to 127.5.118.129:35807: connect: Connection refused (error 111)
I20260620 14:54:18.280484 21494 catalog_manager.cc:4841] TS b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903): tablet 51e74398063541b1861918f7fd461124 (table test-table [id=14b50665d9f84b3697e553f758d1471a]) successfully deleted
I20260620 14:54:18.285908 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 21837
I20260620 14:54:18.292943 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 21475
I20260620 14:54:38.298743 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42153
--webserver_interface=127.5.118.190
--webserver_port=38987
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:42153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:38.387066 21997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:38.387280 21997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:38.387303 21997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:38.388540 21997 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:54:38.388581 21997 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:54:38.388593 21997 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:54:38.388604 21997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:38.388617 21997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:38.388631 21997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:38.388649 21997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:38.388662 21997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:38.390122 21997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:42153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:42153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=38987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:38.390329 21997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:38.390511 21997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:38.395567 22003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:38.395753 21997 server_base.cc:1034] running on GCE node
W20260620 14:54:38.395591 22005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:38.395576 22002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:38.396225 21997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:38.396561 21997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:38.397713 21997 hybrid_clock.cc:648] HybridClock initialized: now 1781967278397692 us; error 35 us; skew 500 ppm
Jun 20 14:54:38 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:38.405786 21997 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:54:38.406327 21997 webserver.cc:460] Webserver started at http://127.5.118.190:38987/ using document root <none> and password file <none>
I20260620 14:54:38.406589 21997 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:38.406641 21997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:38.408165 21997 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:54:38.409083 22012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:38.409300 21997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:38.409411 21997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "fe4b10df2e2a46e582882bdfd04e1a0a"
format_stamp: "Formatted at 2026-06-20 14:53:56 on dist-test-slave-2m5f"
server_key: "6c52e3a5ecfdf0e8218c4cc52c55268f"
server_key_iv: "320bc602e674cbd47514085b7c47e7db"
server_key_version: "encryptionkey@0"
I20260620 14:54:38.409875 21997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:38.424676 21997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:38.426870 21997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:38.427042 21997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:38.431464 21997 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:42153
I20260620 14:54:38.431717 22064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:42153 every 8 connection(s)
I20260620 14:54:38.431942 21997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:38.433490 22065 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:54:38.434144 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Bootstrap starting.
I20260620 14:54:38.434336 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 21997
I20260620 14:54:38.434628 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:35807
--local_ip_for_outbound_sockets=127.5.118.129
--tserver_master_addrs=127.5.118.190:42153
--webserver_port=46689
--webserver_interface=127.5.118.129
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:54:38.442816 22065 log.cc:824] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:38.445755 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:54:38.446187 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Bootstrap complete.
I20260620 14:54:38.448570 22065 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:54:38.448823 22065 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4b10df2e2a46e582882bdfd04e1a0a, State: Initialized, Role: FOLLOWER
I20260620 14:54:38.448999 22065 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:54:38.449082 22065 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:38.449136 22065 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:38.449185 22065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:54:38.450204 22065 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:54:38.450327 22065 leader_election.cc:302] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe4b10df2e2a46e582882bdfd04e1a0a; no voters:
I20260620 14:54:38.450547 22065 leader_election.cc:288] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:54:38.450585 22069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:54:38.450702 22069 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [term 3 LEADER]: Becoming Leader. State: Replica: fe4b10df2e2a46e582882bdfd04e1a0a, State: Running, Role: LEADER
I20260620 14:54:38.450790 22069 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } }
I20260620 14:54:38.450829 22065 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:38.451050 22069 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe4b10df2e2a46e582882bdfd04e1a0a. Latest consensus state: current_term: 3 leader_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } } }
I20260620 14:54:38.451128 22069 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:38.451056 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4b10df2e2a46e582882bdfd04e1a0a" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 42153 } } }
I20260620 14:54:38.451304 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:38.451324 22073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:38.452298 22073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:38.452455 22073 catalog_manager.cc:1259] Loaded cluster ID: 77379055d75f4814aeba65c61cb5e8f6
I20260620 14:54:38.452692 22073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:38.453859 22073 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:38.454046 22073 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Loaded TSK: 0
I20260620 14:54:38.454097 22073 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fe4b10df2e2a46e582882bdfd04e1a0a: Loaded TSK: 1
F20260620 14:54:38.454319 22073 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f20522929ad google::LogMessage::Fail() at ??:0
@ 0x7f205229588a google::LogMessage::SendToLog() at ??:0
@ 0x7f20522924de google::LogMessage::Flush() at ??:0
@ 0x7f2052293a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f20549ff104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f20549fcd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f205a1ec1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f205a1ec48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f205a1ec594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f205a1b2c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f205a1bcad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f20541600d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2054155954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2054fb96db start_thread at ??:0
@ 0x7f2050d9061f clone at ??:0
W20260620 14:54:38.538673 22067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:38.538892 22067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:38.538916 22067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:38.540127 22067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:38.540230 22067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:38.540297 22067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:54:38.541838 22067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:35807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=46689
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:38.542099 22067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:38.542335 22067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:38.547390 22107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:38.547453 22067 server_base.cc:1034] running on GCE node
W20260620 14:54:38.547391 22109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:38.547640 22106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:38.547873 22067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:38.548159 22067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:38.549345 22067 hybrid_clock.cc:648] HybridClock initialized: now 1781967278549328 us; error 34 us; skew 500 ppm
Jun 20 14:54:38 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:38.551874 22067 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:54:38.552454 22067 webserver.cc:460] Webserver started at http://127.5.118.129:46689/ using document root <none> and password file <none>
I20260620 14:54:38.552716 22067 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:38.552762 22067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:38.554148 22067 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:38.554909 22116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:38.555116 22067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:38.555210 22067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "74e2fe2faf24468aa46c9d234c1312f9"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "0c7ea80c8d92a9c7a7417f899b036b69"
server_key_iv: "867a961d514f8b4fca2fd65235148ca9"
server_key_version: "encryptionkey@0"
I20260620 14:54:38.555608 22067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:38.570327 22067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:38.572763 22067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:38.572943 22067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:38.573217 22067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:38.573694 22123 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 14:54:38.575748 22067 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:54:38.575824 22067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 14:54:38.575872 22067 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:54:38.576637 22067 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:54:38.576697 22067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 14:54:38.576779 22123 tablet_bootstrap.cc:492] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: Bootstrap starting.
I20260620 14:54:38.584095 22067 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:35807
I20260620 14:54:38.584542 22067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:38.588483 22230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:35807 every 8 connection(s)
I20260620 14:54:38.588613 22123 log.cc:824] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:38.590087 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 22067
I20260620 14:54:38.590479 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:40119
--local_ip_for_outbound_sockets=127.5.118.130
--tserver_master_addrs=127.5.118.190:42153
--webserver_port=38875
--webserver_interface=127.5.118.130
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:54:38.591639 22123 tablet_bootstrap.cc:492] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:54:38.592018 22123 tablet_bootstrap.cc:492] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: Bootstrap complete.
I20260620 14:54:38.593357 22123 ts_tablet_manager.cc:1397] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260620 14:54:38.594985 22123 raft_consensus.cc:348] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:38.595362 22123 raft_consensus.cc:729] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e2fe2faf24468aa46c9d234c1312f9, State: Initialized, Role: FOLLOWER
I20260620 14:54:38.595567 22123 consensus_queue.cc:260] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:38.595886 22123 ts_tablet_manager.cc:1428] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260620 14:54:38.661100 22233 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:54:38.585318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:49995)
0620 14:54:38.585501 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.585546 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:38.586034 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:38.660899 (+ 74865us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
W20260620 14:54:38.661356 22231 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:42153 (0 consecutive failures): Network error: Failed to ping master at 127.5.118.190:42153: Client connection negotiation failed: client connection to 127.5.118.190:42153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:54:38.661716 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:38.661583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:54677)
0620 14:54:38.661644 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.661685 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:54:38.662168 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:38.662051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:52257)
0620 14:54:38.662127 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.662152 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 14:54:38.662305 22231 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:54:38.676928 22235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:38.677150 22235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:38.677175 22235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:38.678318 22235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:38.678359 22235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:38.678386 22235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:54:38.679793 22235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:40119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=38875
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:38.679986 22235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:38.680233 22235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:38.685264 22241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:38.685330 22242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:38.685503 22235 server_base.cc:1034] running on GCE node
W20260620 14:54:38.685262 22244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:38.685730 22235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:38.685940 22235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:38.687119 22235 hybrid_clock.cc:648] HybridClock initialized: now 1781967278687112 us; error 36 us; skew 500 ppm
Jun 20 14:54:38 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:38.689682 22235 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:54:38.690196 22235 webserver.cc:460] Webserver started at http://127.5.118.130:38875/ using document root <none> and password file <none>
I20260620 14:54:38.690446 22235 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:38.690516 22235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:38.691937 22235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:38.692677 22251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:38.692849 22235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:54:38.692943 22235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "feb35f8c90244891a6bb1893ebf3a925"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "ee64bcc7a597a4a98494249cf3032c88"
server_key_iv: "9c64f2defad8ce19a8710d0b7b283a98"
server_key_version: "encryptionkey@0"
I20260620 14:54:38.693343 22235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:38.703076 22235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:38.705268 22235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:38.705448 22235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:38.705694 22235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:38.706014 22235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:38.706070 22235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:38.706116 22235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:38.706146 22235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:38.712527 22235 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:40119
I20260620 14:54:38.712586 22364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:40119 every 8 connection(s)
I20260620 14:54:38.712978 22235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:38.714170 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:38.713759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:60935)
0620 14:54:38.713945 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.714023 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
W20260620 14:54:38.714648 22365 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:42153 (0 consecutive failures): Network error: Failed to ping master at 127.5.118.190:42153: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
I20260620 14:54:38.714977 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:38.714851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:42853)
0620 14:54:38.714920 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.714954 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:54:38.715301 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:38.715207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:49979)
0620 14:54:38.715253 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.715281 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 14:54:38.715512 22365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:54:38.717248 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 22235
I20260620 14:54:38.717542 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:38903
--local_ip_for_outbound_sockets=127.5.118.131
--tserver_master_addrs=127.5.118.190:42153
--webserver_port=41141
--webserver_interface=127.5.118.131
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:38.802229 22368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:38.802459 22368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:38.802486 22368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:38.803766 22368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:38.803812 22368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:38.803843 22368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:54:38.805423 22368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:38903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=41141
--tserver_master_addrs=127.5.118.190:42153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:38.805688 22368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:38.805949 22368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:38.811192 22376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:38.811200 22374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:38.811205 22373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:38.811623 22368 server_base.cc:1034] running on GCE node
I20260620 14:54:38.811915 22368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:38.812248 22368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:38.813426 22368 hybrid_clock.cc:648] HybridClock initialized: now 1781967278813408 us; error 35 us; skew 500 ppm
Jun 20 14:54:38 dist-test-slave-2m5f krb5kdc[20927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:38.816277 22368 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:54:38.816785 22368 webserver.cc:460] Webserver started at http://127.5.118.131:41141/ using document root <none> and password file <none>
I20260620 14:54:38.817058 22368 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:38.817122 22368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:38.818583 22368 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:38.819322 22383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:38.819682 22368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:38.819824 22368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "b5b46f6ba9cb4316941f10e912543721"
format_stamp: "Formatted at 2026-06-20 14:53:57 on dist-test-slave-2m5f"
server_key: "c5dfbd8317af0a7c21992a2428755bb0"
server_key_iv: "80a527c584cdab4a20496cd4df32bc02"
server_key_version: "encryptionkey@0"
I20260620 14:54:38.820243 22368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:38.835107 22368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:38.837713 22368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:38.837914 22368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:38.838177 22368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:38.838510 22368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:38.838565 22368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:38.838620 22368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:38.838644 22368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:38.845059 22368 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:38903
I20260620 14:54:38.845114 22496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:38903 every 8 connection(s)
I20260620 14:54:38.845484 22368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:38.846652 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:38.846218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:47717)
0620 14:54:38.846381 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.846459 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
W20260620 14:54:38.847059 22497 heartbeater.cc:644] Failed to heartbeat to 127.5.118.190:42153 (0 consecutive failures): Network error: Failed to ping master at 127.5.118.190:42153: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
I20260620 14:54:38.847318 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:38.847249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:49341)
0620 14:54:38.847277 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.847303 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260620 14:54:38.847669 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:38.847573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:32809)
0620 14:54:38.847634 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:38.847655 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:54:38.847785 22497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:54:38.852738 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 22368
I20260620 14:54:39.662868 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:39.662626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:34247)
0620 14:54:39.662781 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:39.662815 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260620 14:54:39.716022 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:39.715806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:43161)
0620 14:54:39.715926 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:39.715967 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260620 14:54:39.848430 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:39.848152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:38505)
0620 14:54:39.848340 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:39.848379 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":123,"threads_started":1}
I20260620 14:54:40.141080 22504 raft_consensus.cc:482] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:40.141206 22504 raft_consensus.cc:504] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:40.141476 22504 leader_election.cc:288] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903), feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119)
Jun 20 14:54:40 dist-test-slave-2m5f krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.131@KRBTEST.COM
Jun 20 14:54:40 dist-test-slave-2m5f krb5kdc[20927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967278, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.130@KRBTEST.COM
I20260620 14:54:40.152153 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:40.142116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:47171 (local address 127.5.118.131:38903)
0620 14:54:40.143038 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:40.143046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:40.143072 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:54:40.143464 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:40.143466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:40.143507 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:40.143640 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:40.143973 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.144853 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.145703 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.146104 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.146217 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:40.148398 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:40.148403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:40.148403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:40.148413 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:40.150291 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:40.150821 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:40.150824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:40.150825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:40.150949 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:40.151241 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:40.151243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:40.151243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:40.151512 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:40.151609 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:40.151900 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 14:54:40.152038 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":63,"threads_started":1}
I20260620 14:54:40.152143 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:40.141758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:38903 (local address 127.5.118.129:47171)
0620 14:54:40.141930 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:40.141943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:40.142011 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:40.143633 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:40.143634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:40.143819 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:40.143866 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:40.143868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.144961 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.144962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:40.145604 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:40.145606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.146202 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.146203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:40.146314 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:40.146819 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:40.146827 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:40.148288 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:40.150398 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:40.150405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:40.150408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:40.150740 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:40.151062 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:40.151063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:40.151063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:40.151165 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:40.151588 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:40.151595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:40.151726 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 14:54:40.151834 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:54:40.152032 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260620 14:54:40.152390 22505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:40.141878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:40119 (local address 127.5.118.129:36733)
0620 14:54:40.142105 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:40.142115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:40.142172 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:40.142997 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:40.143000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:40.143300 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:40.143558 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:40.143565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.144811 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.144812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:40.145615 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:40.145618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.146240 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.146241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:40.146315 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:40.146987 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:40.146992 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:40.148658 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:40.150581 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:40.150582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:40.150583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:40.150831 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:40.151122 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:40.151123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:40.151124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:40.151190 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:40.151593 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:40.151601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:40.151726 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 14:54:40.151835 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:54:40.152115 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":139,"thread_start_us":68,"threads_started":1}
I20260620 14:54:40.152179 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:40.142114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:36733 (local address 127.5.118.130:40119)
0620 14:54:40.142248 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:40.142258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:40.142285 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:54:40.142732 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:40.142734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:40.142778 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:40.142894 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:40.143703 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.144701 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.145729 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:40.146142 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:40.146270 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:40.148774 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:40.148778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:40.148778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:40.148793 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:40.150370 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:40.150926 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:40.150929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:40.150930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:40.151061 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:40.151261 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:40.151262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:40.151262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:40.151512 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:40.151610 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:40.151895 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:54:40.152064 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
W20260620 14:54:40.152987 22119 leader_election.cc:341] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903): Not found: Tablet not found: 51e74398063541b1861918f7fd461124
W20260620 14:54:40.153214 22119 leader_election.cc:341] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119): Not found: Tablet not found: 51e74398063541b1861918f7fd461124
I20260620 14:54:40.153251 22119 leader_election.cc:302] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74e2fe2faf24468aa46c9d234c1312f9; no voters: b5b46f6ba9cb4316941f10e912543721, feb35f8c90244891a6bb1893ebf3a925
I20260620 14:54:40.153373 22504 raft_consensus.cc:2726] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:54:40.663656 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:40.663421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:55903)
0620 14:54:40.663569 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:40.663610 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260620 14:54:40.716871 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:40.716617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:46947)
0620 14:54:40.716759 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:40.716797 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260620 14:54:40.849141 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:40.848910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:46585)
0620 14:54:40.849051 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:40.849099 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260620 14:54:41.664587 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:41.664326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:59915)
0620 14:54:41.664482 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:41.664537 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260620 14:54:41.717679 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:41.717351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:34973)
0620 14:54:41.717586 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:41.717634 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260620 14:54:41.807313 22513 raft_consensus.cc:482] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:41.807425 22513 raft_consensus.cc:504] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:41.807627 22513 leader_election.cc:288] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903), feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119)
W20260620 14:54:41.808215 22119 leader_election.cc:341] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903): Not found: Tablet not found: 51e74398063541b1861918f7fd461124
W20260620 14:54:41.808429 22119 leader_election.cc:341] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119): Not found: Tablet not found: 51e74398063541b1861918f7fd461124
I20260620 14:54:41.808580 22119 leader_election.cc:302] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74e2fe2faf24468aa46c9d234c1312f9; no voters: b5b46f6ba9cb4316941f10e912543721, feb35f8c90244891a6bb1893ebf3a925
I20260620 14:54:41.808653 22513 raft_consensus.cc:2726] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:54:41.849892 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:41.849669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:49847)
0620 14:54:41.849803 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:41.849847 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260620 14:54:42.665396 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:42.665128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:59101)
0620 14:54:42.665301 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:42.665348 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
I20260620 14:54:42.718634 22516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:42.718288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:59099)
0620 14:54:42.718460 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:42.718585 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
I20260620 14:54:42.850692 22517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:42.850424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:53401)
0620 14:54:42.850608 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:42.850645 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":95,"threads_started":1}
I20260620 14:54:43.666379 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:43.666121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:49799)
0620 14:54:43.666286 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:43.666331 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260620 14:54:43.719583 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:43.719248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:50295)
0620 14:54:43.719412 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:43.719462 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260620 14:54:43.851408 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:43.851191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:58647)
0620 14:54:43.851321 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:43.851360 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260620 14:54:44.201515 22521 raft_consensus.cc:482] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:44.201617 22521 raft_consensus.cc:504] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:44.201838 22521 leader_election.cc:288] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903), feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119)
W20260620 14:54:44.202225 22119 leader_election.cc:341] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119): Not found: Tablet not found: 51e74398063541b1861918f7fd461124
W20260620 14:54:44.202329 22119 leader_election.cc:341] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903): Not found: Tablet not found: 51e74398063541b1861918f7fd461124
I20260620 14:54:44.202417 22119 leader_election.cc:302] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74e2fe2faf24468aa46c9d234c1312f9; no voters: b5b46f6ba9cb4316941f10e912543721, feb35f8c90244891a6bb1893ebf3a925
I20260620 14:54:44.202517 22521 raft_consensus.cc:2726] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:54:44.667279 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:44.666966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:53777)
0620 14:54:44.667142 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:44.667191 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":75,"threads_started":1}
I20260620 14:54:44.720423 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:44.720138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:53227)
0620 14:54:44.720331 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:44.720376 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":133,"threads_started":1}
I20260620 14:54:44.852136 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:44.851902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:47949)
0620 14:54:44.852035 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:44.852078 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260620 14:54:45.668076 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:45.667864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:43783)
0620 14:54:45.667990 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:45.668030 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260620 14:54:45.721225 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:45.720952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:34815)
0620 14:54:45.721141 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:45.721181 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260620 14:54:45.852952 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:45.852734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:48983)
0620 14:54:45.852864 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:45.852906 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260620 14:54:46.668819 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:46.668598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:54799)
0620 14:54:46.668726 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:46.668770 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260620 14:54:46.711444 22529 raft_consensus.cc:482] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:46.711607 22529 raft_consensus.cc:504] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:46.711812 22529 leader_election.cc:288] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903), feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119)
W20260620 14:54:46.712379 22119 leader_election.cc:341] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903): Not found: Tablet not found: 51e74398063541b1861918f7fd461124
W20260620 14:54:46.712476 22119 leader_election.cc:341] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119): Not found: Tablet not found: 51e74398063541b1861918f7fd461124
I20260620 14:54:46.712522 22119 leader_election.cc:302] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74e2fe2faf24468aa46c9d234c1312f9; no voters: b5b46f6ba9cb4316941f10e912543721, feb35f8c90244891a6bb1893ebf3a925
I20260620 14:54:46.712666 22529 raft_consensus.cc:2726] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:54:46.721967 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:46.721746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:53747)
0620 14:54:46.721875 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:46.721920 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260620 14:54:46.853946 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:46.853661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:35607)
0620 14:54:46.853830 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:46.853880 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260620 14:54:47.669682 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:47.669428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:60753)
0620 14:54:47.669587 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:47.669634 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260620 14:54:47.722697 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:47.722474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:35871)
0620 14:54:47.722602 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:47.722648 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260620 14:54:47.854734 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:47.854475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:52587)
0620 14:54:47.854641 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:47.854688 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":75,"threads_started":1}
I20260620 14:54:48.670639 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:48.670394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:40795)
0620 14:54:48.670543 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:48.670590 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260620 14:54:48.723512 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:48.723218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:60797)
0620 14:54:48.723388 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:48.723455 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260620 14:54:48.855621 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:48.855352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:49747)
0620 14:54:48.855522 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:48.855572 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260620 14:54:49.330034 22538 raft_consensus.cc:482] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:49.330262 22538 raft_consensus.cc:504] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5b46f6ba9cb4316941f10e912543721" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 38903 } } peers { permanent_uuid: "feb35f8c90244891a6bb1893ebf3a925" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40119 } } peers { permanent_uuid: "74e2fe2faf24468aa46c9d234c1312f9" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 35807 } }
I20260620 14:54:49.330461 22538 leader_election.cc:288] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903), feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119)
W20260620 14:54:49.331076 22119 leader_election.cc:341] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb35f8c90244891a6bb1893ebf3a925 (127.5.118.130:40119): Not found: Tablet not found: 51e74398063541b1861918f7fd461124
W20260620 14:54:49.331149 22119 leader_election.cc:341] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5b46f6ba9cb4316941f10e912543721 (127.5.118.131:38903): Not found: Tablet not found: 51e74398063541b1861918f7fd461124
I20260620 14:54:49.331169 22119 leader_election.cc:302] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 74e2fe2faf24468aa46c9d234c1312f9; no voters: b5b46f6ba9cb4316941f10e912543721, feb35f8c90244891a6bb1893ebf3a925
I20260620 14:54:49.331291 22538 raft_consensus.cc:2726] T 51e74398063541b1861918f7fd461124 P 74e2fe2faf24468aa46c9d234c1312f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:54:49.671561 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:49.671316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:43599)
0620 14:54:49.671466 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:49.671513 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260620 14:54:49.724574 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:49.724152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:40013)
0620 14:54:49.724468 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:49.724523 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":206,"threads_started":1}
I20260620 14:54:49.856416 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:49.856123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:35143)
0620 14:54:49.856328 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:49.856368 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":128,"threads_started":1}
I20260620 14:54:50.672501 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:50.672267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:50671)
0620 14:54:50.672409 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:50.672456 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260620 14:54:50.725502 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:50.725239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:37901)
0620 14:54:50.725396 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:50.725444 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260620 14:54:50.857221 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:50.856966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:39263)
0620 14:54:50.857123 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:50.857168 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
I20260620 14:54:51.673424 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:51.673058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:41329)
0620 14:54:51.673306 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:51.673360 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":115,"threads_started":1}
I20260620 14:54:51.726336 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:51.726062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:52689)
0620 14:54:51.726235 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:51.726288 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":74,"threads_started":1}
I20260620 14:54:51.858006 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:51.857747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:42241)
0620 14:54:51.857912 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:51.857956 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260620 14:54:52.674314 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:52.674030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:37777)
0620 14:54:52.674209 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:52.674266 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
I20260620 14:54:52.727281 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:52.727026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:37417)
0620 14:54:52.727192 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:52.727231 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260620 14:54:52.858790 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:52.858563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:58131)
0620 14:54:52.858700 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:52.858741 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260620 14:54:53.675179 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:53.674944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.129:44141)
0620 14:54:53.675086 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:53.675131 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260620 14:54:53.728159 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:53.727918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.130:38935)
0620 14:54:53.728069 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:53.728113 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260620 14:54:53.857393 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 22067
I20260620 14:54:53.859627 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:53.859409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:42153 (local address 127.5.118.131:42773)
0620 14:54:53.859546 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:53.859586 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.190:42153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260620 14:54:53.862412 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 22235
I20260620 14:54:53.866804 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 22368
2026-06-20T14:54:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58188 ms)
[----------] 30 tests from SecurityITest (240008 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:54:53 dist-test-slave-2m5f krb5kdc[22556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:54:53 dist-test-slave-2m5f krb5kdc[22556](info): set up 2 sockets
Jun 20 14:54:53 dist-test-slave-2m5f krb5kdc[22556](info): commencing operation
krb5kdc: starting...
W20260620 14:54:54.912161 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:54:54 dist-test-slave-2m5f krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:54:55.040732 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:40559
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:40559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:55.127913 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:55.128155 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:55.128252 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:55.129607 22569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:55.129657 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:55.129674 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:55.129689 22569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:55.129703 22569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:55.131552 22569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:40559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:40559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:55.131894 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:55.132220 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:55.137935 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:55.138013 22569 server_base.cc:1034] running on GCE node
W20260620 14:54:55.137935 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:55.138403 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:55.138703 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:55.139063 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:55.140266 22569 hybrid_clock.cc:648] HybridClock initialized: now 1781967295140249 us; error 49 us; skew 500 ppm
Jun 20 14:54:55 dist-test-slave-2m5f krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967295, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:55.143021 22569 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:54:55.143533 22569 webserver.cc:460] Webserver started at http://127.5.118.190:33511/ using document root <none> and password file <none>
I20260620 14:54:55.143786 22569 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:55.143862 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:55.144017 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:55.145232 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "be014e3018d54fd29370a581479a5388"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "c399bd7b8e8c84e892861645b1abc612"
server_key_iv: "3a8cc03af10fa2837d9e06e7bcc08e4e"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.145648 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "be014e3018d54fd29370a581479a5388"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "c399bd7b8e8c84e892861645b1abc612"
server_key_iv: "3a8cc03af10fa2837d9e06e7bcc08e4e"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.147176 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:54:55.148053 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.148289 22569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:55.148408 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "be014e3018d54fd29370a581479a5388"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "c399bd7b8e8c84e892861645b1abc612"
server_key_iv: "3a8cc03af10fa2837d9e06e7bcc08e4e"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.148497 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:55.156468 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:55.158943 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:55.159125 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:55.163138 22569 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:40559
I20260620 14:54:55.163194 22636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:40559 every 8 connection(s)
I20260620 14:54:55.163584 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:55.164587 22637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:55.166150 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 22569
I20260620 14:54:55.166316 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:54:55.166540 5594 external_mini_cluster.cc:1428] Setting key e9b39751a4a6aec2b8ac3c6f9b81ec38
I20260620 14:54:55.167322 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388: Bootstrap starting.
I20260620 14:54:55.167932 22637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:55.168310 22637 log.cc:824] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:55.169061 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388: No bootstrap required, opened a new log
I20260620 14:54:55.170917 22637 raft_consensus.cc:348] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be014e3018d54fd29370a581479a5388" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40559 } }
I20260620 14:54:55.171093 22637 raft_consensus.cc:374] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:55.171139 22637 raft_consensus.cc:729] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be014e3018d54fd29370a581479a5388, State: Initialized, Role: FOLLOWER
I20260620 14:54:55.171320 22637 consensus_queue.cc:260] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be014e3018d54fd29370a581479a5388" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40559 } }
I20260620 14:54:55.171429 22637 raft_consensus.cc:388] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:55.171468 22637 raft_consensus.cc:482] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:55.171530 22637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:55.172411 22637 raft_consensus.cc:504] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be014e3018d54fd29370a581479a5388" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40559 } }
I20260620 14:54:55.172544 22637 leader_election.cc:302] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be014e3018d54fd29370a581479a5388; no voters:
I20260620 14:54:55.172747 22637 leader_election.cc:288] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:55.172851 22642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:55.173066 22642 raft_consensus.cc:686] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [term 1 LEADER]: Becoming Leader. State: Replica: be014e3018d54fd29370a581479a5388, State: Running, Role: LEADER
I20260620 14:54:55.173070 22637 sys_catalog.cc:564] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:55.173175 22642 consensus_queue.cc:237] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be014e3018d54fd29370a581479a5388" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40559 } }
I20260620 14:54:55.173609 22642 sys_catalog.cc:455] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be014e3018d54fd29370a581479a5388. Latest consensus state: current_term: 1 leader_uuid: "be014e3018d54fd29370a581479a5388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be014e3018d54fd29370a581479a5388" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40559 } } }
Jun 20 14:54:55 dist-test-slave-2m5f krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:54:55.173779 22643 sys_catalog.cc:455] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be014e3018d54fd29370a581479a5388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be014e3018d54fd29370a581479a5388" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40559 } } }
I20260620 14:54:55.173960 22643 sys_catalog.cc:458] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:55.174032 22642 sys_catalog.cc:458] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388 [sys.catalog]: This master's current role is: LEADER
W20260620 14:54:55.175081 22658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:54:55.175161 22658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:54:55.175218 22652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:55.175374 22652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:55.177305 22652 catalog_manager.cc:1347] Generated new cluster ID: 287e6fe0cdb2439fa6fa3e310ed743ef
I20260620 14:54:55.177358 22652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:55.178558 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.167110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.5.118.190:40559)
0620 14:54:55.167289 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.167297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.167322 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:54:55.167814 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.167821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.167874 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.168088 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:55.168824 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.169815 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.170658 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.171090 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.171216 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.174490 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:55.174495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:55.174495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:55.174509 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:55.176588 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:55.177047 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:55.177051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:55.177051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:55.177151 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:55.177389 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:55.177390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:55.177391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:55.177715 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:55.177877 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.178211 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.178346 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260620 14:54:55.195827 22652 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:55.196530 22652 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:55.200904 22652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388: Generated new TSK 0
I20260620 14:54:55.201118 22652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:54:55.235214 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40559
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:55.321198 22663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:55.321429 22663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:55.321456 22663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:55.322655 22663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:55.322703 22663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:55.322731 22663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:54:55.324364 22663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:55.324620 22663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:55.324877 22663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:55.330255 22668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:55.330250 22671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:55.330289 22669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:55.330485 22663 server_base.cc:1034] running on GCE node
I20260620 14:54:55.330770 22663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:55.331036 22663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:55.332224 22663 hybrid_clock.cc:648] HybridClock initialized: now 1781967295332182 us; error 59 us; skew 500 ppm
Jun 20 14:54:55 dist-test-slave-2m5f krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967295, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:55.335107 22663 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:54:55.335609 22663 webserver.cc:460] Webserver started at http://127.5.118.129:40003/ using document root <none> and password file <none>
I20260620 14:54:55.335845 22663 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:55.335894 22663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:55.336068 22663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:55.337214 22663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "9e8dd508a5465f663ba6b5537763a7a4"
server_key_iv: "8bf2aa5c5ea5c33adac6d96f3a5229b0"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.337601 22663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "9e8dd508a5465f663ba6b5537763a7a4"
server_key_iv: "8bf2aa5c5ea5c33adac6d96f3a5229b0"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.338991 22663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:54:55.339821 22678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.340075 22663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:55.340210 22663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "9e8dd508a5465f663ba6b5537763a7a4"
server_key_iv: "8bf2aa5c5ea5c33adac6d96f3a5229b0"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.340306 22663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:55.360606 22663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:55.363175 22663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:55.363314 22663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:55.363580 22663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:55.363935 22663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:55.363999 22663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.364037 22663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:55.364089 22663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.369992 22663 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:45683
I20260620 14:54:55.370069 22791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:45683 every 8 connection(s)
I20260620 14:54:55.370450 22663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:55.370733 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 22663
I20260620 14:54:55.370956 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:55.371196 5594 external_mini_cluster.cc:1428] Setting key b4a7ff228f6c754c118c9f795d498d8e
Jun 20 14:54:55 dist-test-slave-2m5f krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967295, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
I20260620 14:54:55.382318 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.371405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:50153 (local address 127.5.118.190:40559)
0620 14:54:55.371464 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.371466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.371696 (+ 230us) server_negotiation.cc:407] Connection header received
0620 14:54:55.372385 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.372387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.372399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.372477 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:55.373591 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.374472 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.376224 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.376480 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.376543 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.379232 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:55.379234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:55.379234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:55.379242 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:55.380799 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:55.381261 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:55.381263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:55.381263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:55.381331 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:55.381604 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:55.381605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:55.381605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:55.381725 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:55.381821 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.382108 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.382207 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:54:55.382508 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.371324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40559 (local address 127.5.118.129:50153)
0620 14:54:55.371568 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.371615 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:55.372245 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:55.372538 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:55.372541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:55.373253 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:55.373485 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.373491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.374587 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.374588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.376045 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.376049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.376583 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.376585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.376744 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.377316 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:55.377329 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:55.379117 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:55.380898 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:55.380906 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:55.380909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:55.381167 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:55.381412 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:55.381413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:55.381413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:55.381535 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:55.381813 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:55.381821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:55.381974 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 14:54:55.382090 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:54:55.382263 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
I20260620 14:54:55.383160 22792 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40559
I20260620 14:54:55.383280 22792 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:55.383510 22792 heartbeater.cc:511] Master 127.5.118.190:40559 requested a full tablet report, sending...
I20260620 14:54:55.384093 22601 ts_manager.cc:194] Registered new tserver with Master: 0b9dc358d5b144b19ba8a5b6232eb6d6 (127.5.118.129:45683)
I20260620 14:54:55.384829 22601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:50153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:54:55.399454 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40559
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:55.482131 22797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:55.482367 22797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:55.482393 22797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:55.483620 22797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:55.483671 22797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:55.483700 22797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:54:55.485304 22797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:55.485584 22797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:55.485846 22797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:55.491065 22802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:55.491065 22805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:55.491118 22803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:55.491415 22797 server_base.cc:1034] running on GCE node
I20260620 14:54:55.491632 22797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:55.491912 22797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:55.493055 22797 hybrid_clock.cc:648] HybridClock initialized: now 1781967295493025 us; error 51 us; skew 500 ppm
Jun 20 14:54:55 dist-test-slave-2m5f krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967295, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:55.495795 22797 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:54:55.496366 22797 webserver.cc:460] Webserver started at http://127.5.118.130:36849/ using document root <none> and password file <none>
I20260620 14:54:55.496563 22797 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:55.496604 22797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:55.496697 22797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:55.497849 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "85a27f189ff64251a0b3bee4bd5fe13f"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "53bcb8db2b5bac4c4291f3c042524c47"
server_key_iv: "d6994597b89f560f76488c70b92fe54e"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.498160 22797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "85a27f189ff64251a0b3bee4bd5fe13f"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "53bcb8db2b5bac4c4291f3c042524c47"
server_key_iv: "d6994597b89f560f76488c70b92fe54e"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.499481 22797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:54:55.500335 22812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.500599 22797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:54:55.500710 22797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "85a27f189ff64251a0b3bee4bd5fe13f"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "53bcb8db2b5bac4c4291f3c042524c47"
server_key_iv: "d6994597b89f560f76488c70b92fe54e"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.500790 22797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:55.510404 22797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:55.513409 22797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:55.513571 22797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:55.513833 22797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:55.514164 22797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:55.514199 22797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.514223 22797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:55.514269 22797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.520121 22797 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:44763
I20260620 14:54:55.520237 22925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:44763 every 8 connection(s)
I20260620 14:54:55.520571 22797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:55.525172 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 22797
I20260620 14:54:55.525277 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:55.525521 5594 external_mini_cluster.cc:1428] Setting key 799692f10171866668bbd9ea6878666d
Jun 20 14:54:55 dist-test-slave-2m5f krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967295, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:54:55.531273 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.521470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:39041 (local address 127.5.118.190:40559)
0620 14:54:55.521565 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.521567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.521838 (+ 271us) server_negotiation.cc:407] Connection header received
0620 14:54:55.522393 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.522395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.522408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.522480 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:55.523503 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.524027 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.525172 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.525477 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.525566 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.528200 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:55.528202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:55.528202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:55.528211 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:55.529836 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:55.530252 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:55.530253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:55.530254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:55.530328 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:55.530580 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:55.530582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:55.530582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:55.530717 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:55.530799 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.531093 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.531179 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:54:55.531476 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.521341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40559 (local address 127.5.118.130:39041)
0620 14:54:55.521603 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.521659 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:55.522282 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:55.522568 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:55.522572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:55.523165 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:55.523392 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.523397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.524136 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.524137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.525067 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.525069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.525580 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.525581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.525727 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.526361 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:55.526370 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:55.528077 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:55.529931 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:55.529937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:55.529939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:55.530172 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:55.530398 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:55.530399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:55.530399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:55.530499 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:55.530789 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:55.530797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:55.530943 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 14:54:55.531055 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:54:55.531226 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":115,"threads_started":1}
I20260620 14:54:55.532100 22926 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40559
I20260620 14:54:55.532269 22926 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:55.532465 22926 heartbeater.cc:511] Master 127.5.118.190:40559 requested a full tablet report, sending...
I20260620 14:54:55.532933 22601 ts_manager.cc:194] Registered new tserver with Master: 85a27f189ff64251a0b3bee4bd5fe13f (127.5.118.130:44763)
I20260620 14:54:55.533301 22601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:39041
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:54:55.554466 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40559
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:55.641453 22931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:55.641681 22931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:55.641709 22931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:55.643049 22931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:55.643091 22931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:55.643121 22931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:54:55.644804 22931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:55.645061 22931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:55.645313 22931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:55.650660 22939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:55.650692 22931 server_base.cc:1034] running on GCE node
W20260620 14:54:55.650665 22937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:55.650662 22936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:55.651067 22931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:55.651342 22931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:55.652522 22931 hybrid_clock.cc:648] HybridClock initialized: now 1781967295652517 us; error 43 us; skew 500 ppm
Jun 20 14:54:55 dist-test-slave-2m5f krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967295, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:55.655290 22931 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:54:55.655700 22931 webserver.cc:460] Webserver started at http://127.5.118.131:38049/ using document root <none> and password file <none>
I20260620 14:54:55.655934 22931 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:55.655982 22931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:55.656118 22931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:55.657217 22931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "dc22c6fcdde14fb7947bc127c8e3d973"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "2b24412990420b3b5b797cf0784db185"
server_key_iv: "29fa322266fe0c8fe7340bfb9de5d3ca"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.657590 22931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "dc22c6fcdde14fb7947bc127c8e3d973"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "2b24412990420b3b5b797cf0784db185"
server_key_iv: "29fa322266fe0c8fe7340bfb9de5d3ca"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.658980 22931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:54:55.659852 22946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.660104 22931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:55.660236 22931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "dc22c6fcdde14fb7947bc127c8e3d973"
format_stamp: "Formatted at 2026-06-20 14:54:55 on dist-test-slave-2m5f"
server_key: "2b24412990420b3b5b797cf0784db185"
server_key_iv: "29fa322266fe0c8fe7340bfb9de5d3ca"
server_key_version: "encryptionkey@0"
I20260620 14:54:55.660377 22931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:55.683144 22931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:55.685765 22931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:55.685897 22931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:55.686195 22931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:55.686561 22931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:55.686621 22931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.686664 22931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:55.686691 22931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:55.693022 22931 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:40545
I20260620 14:54:55.693109 23059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:40545 every 8 connection(s)
I20260620 14:54:55.693442 22931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:54:55 dist-test-slave-2m5f krb5kdc[22556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967295, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:54:55.702823 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 22931
I20260620 14:54:55.702936 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:54:55.703167 5594 external_mini_cluster.cc:1428] Setting key 010e6b03ba682111715356da52679baf
I20260620 14:54:55.704085 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.694228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:53945 (local address 127.5.118.190:40559)
0620 14:54:55.694289 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.694291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.694517 (+ 226us) server_negotiation.cc:407] Connection header received
0620 14:54:55.695502 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.695503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.695515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.695584 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:55.696299 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.696840 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.698014 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.698214 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.698288 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.700914 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:55.700915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:55.700915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:55.700922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:55.702408 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:55.702844 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:55.702846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:55.702847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:55.702920 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:55.703256 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:55.703258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:55.703258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:55.703427 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:55.703503 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.703925 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.704020 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 14:54:55.704384 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.694214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40559 (local address 127.5.118.131:53945)
0620 14:54:55.694375 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.694421 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:55.695384 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:55.695643 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:55.695646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:55.695915 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:55.696118 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.696123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.696970 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.696971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.697872 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.697874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.698307 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.698308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.698436 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.699087 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:55.699097 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:55.700791 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:55.702496 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:55.702509 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:55.702511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:55.702746 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:55.703019 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:55.703020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:55.703021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:55.703176 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:55.703552 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:55.703558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:55.703723 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 14:54:55.703894 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 14:54:55.704109 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260620 14:54:55.704873 23060 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40559
I20260620 14:54:55.704977 23060 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:55.705133 23060 heartbeater.cc:511] Master 127.5.118.190:40559 requested a full tablet report, sending...
I20260620 14:54:55.705515 22601 ts_manager.cc:194] Registered new tserver with Master: dc22c6fcdde14fb7947bc127c8e3d973 (127.5.118.131:40545)
I20260620 14:54:55.705899 22601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:53945
I20260620 14:54:55.715577 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:54:55.723574 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.716973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.5.118.190:40559)
0620 14:54:55.717059 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.717061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.717116 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:54:55.717170 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.717171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.717181 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.717232 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:55.717857 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.718394 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.719215 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.719421 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.719484 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.720941 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:55.720942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:55.720943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:55.720950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:55.722430 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:55.722771 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:55.722772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:55.722772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:55.722822 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:55.723033 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:55.723034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:55.723034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:55.723179 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:55.723259 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.723417 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.723481 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:54:55.728804 22601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:55.730444 22601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:55.737385 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.733143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:40545 (local address 127.0.0.1:56862)
0620 14:54:55.733335 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.733575 (+ 240us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:55.733633 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:55.733902 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:55.733903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:55.733906 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:55.733959 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.733961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.734995 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.734996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.736255 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.736258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.737071 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.737071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.737153 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.737167 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:54:55.737243 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:54:55.737300 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":52,"threads_started":1}
I20260620 14:54:55.737398 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.733318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.5.118.131:40545)
0620 14:54:55.733634 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.733638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.733647 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:54:55.733718 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.733721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.733738 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.733794 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:55.734049 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.734898 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.736346 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.736976 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.737066 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.737131 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.737271 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.737314 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":80,"threads_started":1}
I20260620 14:54:55.737761 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.733151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:45683 (local address 127.0.0.1:55158)
0620 14:54:55.733683 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.733693 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:55.733734 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:55.734116 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:55.734117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:55.734120 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:55.734171 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.734172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.735492 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.735493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.736567 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.736570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.737532 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.737533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.737598 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.737602 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:54:55.737649 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 14:54:55.737689 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":92,"threads_started":1}
I20260620 14:54:55.737983 23072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.733143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:44763 (local address 127.0.0.1:38992)
0620 14:54:55.733432 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.733478 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:55.733547 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:55.733833 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:55.733836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:55.733838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:55.733910 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.733913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.735319 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.735320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.736403 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.736405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.737419 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.737420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.737800 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.737805 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:54:55.737865 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:54:55.737911 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":2,"spinlock_wait_cycles":896,"thread_start_us":103,"threads_started":1}
I20260620 14:54:55.738009 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.733353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.5.118.130:44763)
0620 14:54:55.733529 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.733534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.733547 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:54:55.733666 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.733670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.733687 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.733768 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:55.734007 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.735203 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.736493 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.737240 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.737316 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.737380 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.737886 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.737932 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260620 14:54:55.738173 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.733431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.5.118.129:45683)
0620 14:54:55.733731 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.733735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.733744 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:54:55.733937 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.733941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.733957 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.734031 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:55.734344 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.735393 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.736657 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.737376 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.737443 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.737501 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.737930 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.737975 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":71,"threads_started":1}
I20260620 14:54:55.738333 22994 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1f775d0ba74715b2af62494d51fabf (DEFAULT_TABLE table=test-table [id=b2b49b5b19f84ee0bb15f19bbd7b8e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:55.738677 22860 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1f775d0ba74715b2af62494d51fabf (DEFAULT_TABLE table=test-table [id=b2b49b5b19f84ee0bb15f19bbd7b8e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:55.738758 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1f775d0ba74715b2af62494d51fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:55.738931 22860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1f775d0ba74715b2af62494d51fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:55.738935 22726 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1f775d0ba74715b2af62494d51fabf (DEFAULT_TABLE table=test-table [id=b2b49b5b19f84ee0bb15f19bbd7b8e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:55.739235 22726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1f775d0ba74715b2af62494d51fabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:55.741760 23077 tablet_bootstrap.cc:492] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f: Bootstrap starting.
I20260620 14:54:55.742271 23078 tablet_bootstrap.cc:492] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973: Bootstrap starting.
I20260620 14:54:55.742703 23079 tablet_bootstrap.cc:492] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6: Bootstrap starting.
I20260620 14:54:55.742867 23077 tablet_bootstrap.cc:654] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:55.742995 23078 tablet_bootstrap.cc:654] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:55.743376 23079 tablet_bootstrap.cc:654] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:55.743618 23078 log.cc:824] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:55.743618 23077 log.cc:824] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:55.744041 23079 log.cc:824] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:55.744475 23077 tablet_bootstrap.cc:492] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f: No bootstrap required, opened a new log
I20260620 14:54:55.744477 23078 tablet_bootstrap.cc:492] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973: No bootstrap required, opened a new log
I20260620 14:54:55.744553 23077 ts_tablet_manager.cc:1397] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:54:55.744553 23078 ts_tablet_manager.cc:1397] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:54:55.744882 23079 tablet_bootstrap.cc:492] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6: No bootstrap required, opened a new log
I20260620 14:54:55.744985 23079 ts_tablet_manager.cc:1397] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:54:55.745995 23078 raft_consensus.cc:348] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 44763 } } peers { permanent_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40545 } } peers { permanent_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 45683 } }
I20260620 14:54:55.746170 23078 raft_consensus.cc:374] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:55.746217 23078 raft_consensus.cc:729] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc22c6fcdde14fb7947bc127c8e3d973, State: Initialized, Role: FOLLOWER
I20260620 14:54:55.746332 23078 consensus_queue.cc:260] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 44763 } } peers { permanent_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40545 } } peers { permanent_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 45683 } }
I20260620 14:54:55.746682 23078 ts_tablet_manager.cc:1428] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:54:55.746682 23060 heartbeater.cc:503] Master 127.5.118.190:40559 was elected leader, sending a full tablet report...
I20260620 14:54:55.746600 23077 raft_consensus.cc:348] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 44763 } } peers { permanent_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40545 } } peers { permanent_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 45683 } }
I20260620 14:54:55.746778 23077 raft_consensus.cc:374] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:55.746814 23077 raft_consensus.cc:729] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a27f189ff64251a0b3bee4bd5fe13f, State: Initialized, Role: FOLLOWER
I20260620 14:54:55.746953 23079 raft_consensus.cc:348] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 44763 } } peers { permanent_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40545 } } peers { permanent_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 45683 } }
I20260620 14:54:55.747130 23079 raft_consensus.cc:374] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:55.747169 23079 raft_consensus.cc:729] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9dc358d5b144b19ba8a5b6232eb6d6, State: Initialized, Role: FOLLOWER
I20260620 14:54:55.747181 23077 consensus_queue.cc:260] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 44763 } } peers { permanent_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40545 } } peers { permanent_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 45683 } }
I20260620 14:54:55.747304 23079 consensus_queue.cc:260] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 44763 } } peers { permanent_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40545 } } peers { permanent_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 45683 } }
I20260620 14:54:55.747478 23077 ts_tablet_manager.cc:1428] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:55.747520 22926 heartbeater.cc:503] Master 127.5.118.190:40559 was elected leader, sending a full tablet report...
I20260620 14:54:55.747552 23079 ts_tablet_manager.cc:1428] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 14:54:55.747650 22792 heartbeater.cc:503] Master 127.5.118.190:40559 was elected leader, sending a full tablet report...
W20260620 14:54:55.771198 22927 tablet.cc:2369] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:54:55.825059 23084 raft_consensus.cc:482] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:55.825253 23084 raft_consensus.cc:504] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 44763 } } peers { permanent_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40545 } } peers { permanent_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 45683 } }
I20260620 14:54:55.825654 23084 leader_election.cc:288] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc22c6fcdde14fb7947bc127c8e3d973 (127.5.118.131:40545), 0b9dc358d5b144b19ba8a5b6232eb6d6 (127.5.118.129:45683)
I20260620 14:54:55.828908 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.825974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:33347 (local address 127.5.118.131:40545)
0620 14:54:55.826020 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.826023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.826033 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:54:55.826063 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.826064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.826081 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.826141 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:55.826312 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.826771 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.828039 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.828579 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.828637 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.828644 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.828822 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.828852 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:54:55.828925 22928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.825869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:40545 (local address 127.5.118.130:33347)
0620 14:54:55.825931 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.825944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:55.825993 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:55.826161 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:55.826162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:55.826164 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:55.826240 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.826241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.826912 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.826913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.827933 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.827936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.828660 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.828661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.828741 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.828745 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:54:55.828807 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:54:55.828847 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:54:55.829304 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.825904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:45683 (local address 127.5.118.130:38577)
0620 14:54:55.826082 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.826092 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:55.826131 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:55.826419 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:55.826420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:55.826422 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:55.826484 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.826485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.827261 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.827262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.828323 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:55.828326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.829042 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.829043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:55.829119 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.829122 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:54:55.829198 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:54:55.829235 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":125,"threads_started":1}
I20260620 14:54:55.829317 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.826013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:38577 (local address 127.5.118.129:45683)
0620 14:54:55.826088 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.826090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.826297 (+ 207us) server_negotiation.cc:407] Connection header received
0620 14:54:55.826322 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.826323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.826331 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.826396 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:55.826561 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.827167 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.828408 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.828909 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.828981 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.828990 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.829213 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.829255 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:54:55.829329 23014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1f775d0ba74715b2af62494d51fabf" candidate_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" is_pre_election: true
I20260620 14:54:55.829535 23014 raft_consensus.cc:2454] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a27f189ff64251a0b3bee4bd5fe13f in term 0.
I20260620 14:54:55.829676 22746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1f775d0ba74715b2af62494d51fabf" candidate_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" is_pre_election: true
I20260620 14:54:55.829840 22746 raft_consensus.cc:2454] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a27f189ff64251a0b3bee4bd5fe13f in term 0.
I20260620 14:54:55.829799 22814 leader_election.cc:302] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85a27f189ff64251a0b3bee4bd5fe13f, dc22c6fcdde14fb7947bc127c8e3d973; no voters:
I20260620 14:54:55.829950 23084 raft_consensus.cc:2781] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:55.830039 23084 raft_consensus.cc:482] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:55.830075 23084 raft_consensus.cc:3037] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:55.830799 23084 raft_consensus.cc:504] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 44763 } } peers { permanent_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40545 } } peers { permanent_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 45683 } }
I20260620 14:54:55.830960 23084 leader_election.cc:288] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [CANDIDATE]: Term 1 election: Requested vote from peers dc22c6fcdde14fb7947bc127c8e3d973 (127.5.118.131:40545), 0b9dc358d5b144b19ba8a5b6232eb6d6 (127.5.118.129:45683)
I20260620 14:54:55.831142 22746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1f775d0ba74715b2af62494d51fabf" candidate_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6"
I20260620 14:54:55.831221 22746 raft_consensus.cc:3037] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:55.832052 22746 raft_consensus.cc:2454] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a27f189ff64251a0b3bee4bd5fe13f in term 1.
I20260620 14:54:55.832178 23014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1f775d0ba74715b2af62494d51fabf" candidate_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc22c6fcdde14fb7947bc127c8e3d973"
I20260620 14:54:55.832304 22816 leader_election.cc:302] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b9dc358d5b144b19ba8a5b6232eb6d6, 85a27f189ff64251a0b3bee4bd5fe13f; no voters:
I20260620 14:54:55.832285 23014 raft_consensus.cc:3037] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:55.832406 23084 raft_consensus.cc:2781] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:55.832634 23084 raft_consensus.cc:686] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 1 LEADER]: Becoming Leader. State: Replica: 85a27f189ff64251a0b3bee4bd5fe13f, State: Running, Role: LEADER
I20260620 14:54:55.832820 23084 consensus_queue.cc:237] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 44763 } } peers { permanent_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40545 } } peers { permanent_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 45683 } }
I20260620 14:54:55.833174 23014 raft_consensus.cc:2454] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a27f189ff64251a0b3bee4bd5fe13f in term 1.
I20260620 14:54:55.833739 22600 catalog_manager.cc:5495] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a27f189ff64251a0b3bee4bd5fe13f (127.5.118.130). New cstate: current_term: 1 leader_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a27f189ff64251a0b3bee4bd5fe13f" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 45683 } health_report { overall_health: UNKNOWN } } }
I20260620 14:54:55.856998 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.854349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.5.118.130:44763)
0620 14:54:55.854416 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:55.854418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:55.854435 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:54:55.854471 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:55.854472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:55.854491 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:55.854555 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:55.854744 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.855308 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.856084 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:55.856356 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:55.856414 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:55.856587 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:55.856691 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:55.856757 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:55.856896 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 14:54:55.856917 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:54:55.858964 23014 raft_consensus.cc:1261] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973 [term 1 FOLLOWER]: Refusing update from remote peer 85a27f189ff64251a0b3bee4bd5fe13f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:55.859090 22746 raft_consensus.cc:1261] T 1e1f775d0ba74715b2af62494d51fabf P 0b9dc358d5b144b19ba8a5b6232eb6d6 [term 1 FOLLOWER]: Refusing update from remote peer 85a27f189ff64251a0b3bee4bd5fe13f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:55.859470 23084 consensus_queue.cc:1035] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b9dc358d5b144b19ba8a5b6232eb6d6" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 45683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:55.859656 23087 consensus_queue.cc:1035] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc22c6fcdde14fb7947bc127c8e3d973" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 40545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:55.863071 23092 mvcc.cc:205] Tried to move back new op lower bound from 7298938043835183104 to 7298938043731804160. Current Snapshot: MvccSnapshot[applied={T|T < 7298938043835183104}]
I20260620 14:54:55.864428 22600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:55.864533 22600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:55.865288 22600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be014e3018d54fd29370a581479a5388: Sending DeleteTablet for 3 replicas of tablet 1e1f775d0ba74715b2af62494d51fabf
I20260620 14:54:55.865644 22860 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1f775d0ba74715b2af62494d51fabf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:55 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:38992
I20260620 14:54:55.865688 22994 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1f775d0ba74715b2af62494d51fabf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:55 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:56862
I20260620 14:54:55.865722 22726 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e1f775d0ba74715b2af62494d51fabf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:55 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:55158
I20260620 14:54:55.866227 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 22663
I20260620 14:54:55.872364 23102 tablet_replica.cc:330] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973: stopping tablet replica
I20260620 14:54:55.872676 23102 raft_consensus.cc:2229] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 14:54:55.872701 22588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:45683 (error 108)
I20260620 14:54:55.872911 23102 raft_consensus.cc:2258] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:55.873206 23101 tablet_replica.cc:330] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f: stopping tablet replica
I20260620 14:54:55.873370 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.873247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:45683 (local address 127.0.0.1:55162)
0620 14:54:55.873291 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.873332 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:45683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 14:54:55.873368 23101 raft_consensus.cc:2229] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:55.873489 23101 raft_consensus.cc:2258] T 1e1f775d0ba74715b2af62494d51fabf P 85a27f189ff64251a0b3bee4bd5fe13f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:55.873534 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 22797
W20260620 14:54:55.874454 22588 catalog_manager.cc:4584] TS 0b9dc358d5b144b19ba8a5b6232eb6d6 (127.5.118.129:45683): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e1f775d0ba74715b2af62494d51fabf: Network error: Client connection negotiation failed: client connection to 127.5.118.129:45683: connect: Connection refused (error 111)
I20260620 14:54:55.874485 23102 ts_tablet_manager.cc:1905] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:55.876151 23102 ts_tablet_manager.cc:1918] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:55.876282 23102 log.cc:1196] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/1e1f775d0ba74715b2af62494d51fabf
I20260620 14:54:55.876667 23102 ts_tablet_manager.cc:1939] T 1e1f775d0ba74715b2af62494d51fabf P dc22c6fcdde14fb7947bc127c8e3d973: Deleting consensus metadata
I20260620 14:54:55.877265 22586 catalog_manager.cc:4841] TS dc22c6fcdde14fb7947bc127c8e3d973 (127.5.118.131:40545): tablet 1e1f775d0ba74715b2af62494d51fabf (table test-table [id=b2b49b5b19f84ee0bb15f19bbd7b8e6f]) successfully deleted
I20260620 14:54:55.879642 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 22931
I20260620 14:54:55.879731 23071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:55.879568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:44763 (local address 127.0.0.1:39016)
0620 14:54:55.879632 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:55.879663 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.130:44763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:54:55.885142 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 22569
2026-06-20T14:54:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2018 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T14:54:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:55Z Disabled control of system clock
I20260620 14:54:55.913913 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:41289
--webserver_interface=127.5.118.190
--webserver_port=0
--builtin_ntp_servers=127.5.118.148:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:41289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:54:55.997468 23107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:55.997712 23107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:55.997743 23107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:55.998972 23107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:55.999022 23107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:55.999034 23107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:55.999047 23107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:55.999059 23107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:56.000485 23107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:41289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:56.000703 23107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:56.000989 23107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:56.006233 23115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:56.006263 23113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:56.006263 23107 server_base.cc:1034] running on GCE node
W20260620 14:54:56.006470 23112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:56.006832 23107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:56.007174 23107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:56.008342 23107 hybrid_clock.cc:648] HybridClock initialized: now 1781967296008339 us; error 44 us; skew 500 ppm
I20260620 14:54:56.009685 23107 webserver.cc:460] Webserver started at http://127.5.118.190:39817/ using document root <none> and password file <none>
I20260620 14:54:56.009975 23107 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:56.010051 23107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:56.010202 23107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:56.011302 23107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "599b5b3d64b040cb8ce48eab2a00f5f4"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "1201e92795abf046e84a3f031661f936"
server_key_iv: "02c16be53c963efeef4cbca89f016582"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.011706 23107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "599b5b3d64b040cb8ce48eab2a00f5f4"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "1201e92795abf046e84a3f031661f936"
server_key_iv: "02c16be53c963efeef4cbca89f016582"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.013550 23107 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 14:54:56.014580 23121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.014796 23107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:54:56.014914 23107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "599b5b3d64b040cb8ce48eab2a00f5f4"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "1201e92795abf046e84a3f031661f936"
server_key_iv: "02c16be53c963efeef4cbca89f016582"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.014997 23107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:56.027153 23107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:56.027525 23107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:56.027684 23107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:56.032132 23107 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:41289
I20260620 14:54:56.032166 23173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:41289 every 8 connection(s)
I20260620 14:54:56.032596 23107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:56.033664 23174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:56.036120 23174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4: Bootstrap starting.
I20260620 14:54:56.036774 23174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:56.037084 23174 log.cc:824] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:56.037887 23174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4: No bootstrap required, opened a new log
I20260620 14:54:56.038709 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 23107
I20260620 14:54:56.038831 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:54:56.039072 5594 external_mini_cluster.cc:1428] Setting key 382bc30dbf81da6cc26015293c4bd31c
I20260620 14:54:56.040007 23174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599b5b3d64b040cb8ce48eab2a00f5f4" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41289 } }
I20260620 14:54:56.040213 23174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:56.040264 23174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599b5b3d64b040cb8ce48eab2a00f5f4, State: Initialized, Role: FOLLOWER
I20260620 14:54:56.040462 23174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599b5b3d64b040cb8ce48eab2a00f5f4" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41289 } }
I20260620 14:54:56.040546 23174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:56.040570 23174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:56.040640 23174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:56.041239 23174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599b5b3d64b040cb8ce48eab2a00f5f4" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41289 } }
I20260620 14:54:56.041330 23174 leader_election.cc:302] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 599b5b3d64b040cb8ce48eab2a00f5f4; no voters:
I20260620 14:54:56.041540 23174 leader_election.cc:288] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:56.041576 23179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:56.041682 23179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [term 1 LEADER]: Becoming Leader. State: Replica: 599b5b3d64b040cb8ce48eab2a00f5f4, State: Running, Role: LEADER
I20260620 14:54:56.041765 23179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599b5b3d64b040cb8ce48eab2a00f5f4" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41289 } }
I20260620 14:54:56.041816 23174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:56.042100 23179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 599b5b3d64b040cb8ce48eab2a00f5f4. Latest consensus state: current_term: 1 leader_uuid: "599b5b3d64b040cb8ce48eab2a00f5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599b5b3d64b040cb8ce48eab2a00f5f4" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41289 } } }
I20260620 14:54:56.042112 23180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "599b5b3d64b040cb8ce48eab2a00f5f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599b5b3d64b040cb8ce48eab2a00f5f4" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41289 } } }
I20260620 14:54:56.042177 23179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:56.042225 23180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:56.042459 23184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:56.043011 23184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:56.044663 23184 catalog_manager.cc:1347] Generated new cluster ID: 200f3f69cb3d47cf85a1a941a68c5fd9
I20260620 14:54:56.044723 23184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:56.045521 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.039649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.5.118.190:41289)
0620 14:54:56.039865 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.039872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.039897 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:54:56.039996 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.040001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.040056 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.040216 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:56.040459 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.042029 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.043223 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.043807 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.043922 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.044774 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:56.044780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:56.044780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:56.044983 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:56.045023 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:56.045034 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:56.045112 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.045231 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.045314 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260620 14:54:56.059448 23184 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:56.060114 23184 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:56.072422 23184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4: Generated new TSK 0
I20260620 14:54:56.072674 23184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:54:56.083189 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:41289
--builtin_ntp_servers=127.5.118.148:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:54:56.186335 23198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:56.186612 23198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:56.186651 23198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:56.187986 23198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:56.188066 23198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:56.188110 23198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:54:56.189643 23198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:41289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:56.190052 23198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:56.190397 23198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:56.196465 23206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:56.196458 23203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:56.196465 23204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:56.196966 23198 server_base.cc:1034] running on GCE node
I20260620 14:54:56.197176 23198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:56.197469 23198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:56.198649 23198 hybrid_clock.cc:648] HybridClock initialized: now 1781967296198627 us; error 36 us; skew 500 ppm
I20260620 14:54:56.200456 23198 webserver.cc:460] Webserver started at http://127.5.118.129:40227/ using document root <none> and password file <none>
I20260620 14:54:56.200762 23198 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:56.200822 23198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:56.200984 23198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:56.202248 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "03c30c88af6d49249c1adac5f78979c8"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "b8494d632585c3d79ed7fbf95dd7b625"
server_key_iv: "eaea0eecd5844e41a2500b1995dc5ef5"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.202705 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "03c30c88af6d49249c1adac5f78979c8"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "b8494d632585c3d79ed7fbf95dd7b625"
server_key_iv: "eaea0eecd5844e41a2500b1995dc5ef5"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.204097 23198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:54:56.205030 23212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.205247 23198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:56.205328 23198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "03c30c88af6d49249c1adac5f78979c8"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "b8494d632585c3d79ed7fbf95dd7b625"
server_key_iv: "eaea0eecd5844e41a2500b1995dc5ef5"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.205417 23198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:56.225093 23198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:56.225416 23198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:56.225528 23198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:56.225730 23198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:56.226097 23198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:56.226136 23198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.226188 23198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:56.226212 23198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.232107 23198 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:37335
I20260620 14:54:56.232263 23325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:37335 every 8 connection(s)
I20260620 14:54:56.232646 23198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:56.238658 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.233635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:44845 (local address 127.5.118.190:41289)
0620 14:54:56.233702 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.233704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.234000 (+ 296us) server_negotiation.cc:407] Connection header received
0620 14:54:56.234558 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.234560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.234576 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.234627 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:56.235400 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.235935 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.237066 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.237285 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.237382 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.238325 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:56.238327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:56.238327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:56.238336 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:56.238353 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:56.238364 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:56.238438 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.238536 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.238573 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:54:56.238879 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.233631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:41289 (local address 127.5.118.129:44845)
0620 14:54:56.233815 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:56.233851 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:56.234443 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:56.234720 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:56.234724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:56.234733 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:56.235250 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.235255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.236055 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.236056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.236962 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.236965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.237395 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.237395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.237537 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.238154 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:56.238154 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:56.238185 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:56.238211 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:56.238436 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:56.238444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:56.238447 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:54:56.238538 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:54:56.238648 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260620 14:54:56.239542 23326 heartbeater.cc:347] Connected to a master server at 127.5.118.190:41289
I20260620 14:54:56.239658 23326 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:56.239974 23326 heartbeater.cc:511] Master 127.5.118.190:41289 requested a full tablet report, sending...
I20260620 14:54:56.240104 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 23198
I20260620 14:54:56.240264 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:56.240504 5594 external_mini_cluster.cc:1428] Setting key 926367490fafe9fdb4fdd1d377fd9c0f
I20260620 14:54:56.240633 23138 ts_manager.cc:194] Registered new tserver with Master: 03c30c88af6d49249c1adac5f78979c8 (127.5.118.129:37335)
I20260620 14:54:56.241628 23138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.129:44845
I20260620 14:54:56.241660 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:41289
--builtin_ntp_servers=127.5.118.148:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:54:56.326222 23329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:56.326445 23329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:56.326468 23329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:56.327584 23329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:56.327618 23329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:56.327642 23329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:54:56.329025 23329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:41289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:56.329238 23329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:56.329504 23329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:56.334870 23335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:56.335071 23329 server_base.cc:1034] running on GCE node
W20260620 14:54:56.334862 23337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:56.334862 23334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:56.335536 23329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:56.335786 23329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:56.336915 23329 hybrid_clock.cc:648] HybridClock initialized: now 1781967296336899 us; error 30 us; skew 500 ppm
I20260620 14:54:56.338537 23329 webserver.cc:460] Webserver started at http://127.5.118.130:41473/ using document root <none> and password file <none>
I20260620 14:54:56.338851 23329 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:56.338934 23329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:56.339102 23329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:56.340454 23329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "c687d74d791045838def0c072e5e0bd9"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "1423f44a8c25cd81b9e072890827f698"
server_key_iv: "25282278204ea82314b2a0e870d67901"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.340889 23329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "c687d74d791045838def0c072e5e0bd9"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "1423f44a8c25cd81b9e072890827f698"
server_key_iv: "25282278204ea82314b2a0e870d67901"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.342499 23329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:54:56.343452 23343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.343628 23329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:54:56.343751 23329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "c687d74d791045838def0c072e5e0bd9"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "1423f44a8c25cd81b9e072890827f698"
server_key_iv: "25282278204ea82314b2a0e870d67901"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.343839 23329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:56.366027 23329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:56.366425 23329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:56.366611 23329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:56.366888 23329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:56.367270 23329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:56.367334 23329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.367380 23329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:56.367414 23329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.374025 23329 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:33411
I20260620 14:54:56.374089 23456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:33411 every 8 connection(s)
I20260620 14:54:56.374497 23329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:56.377686 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 23329
I20260620 14:54:56.377940 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:56.378188 5594 external_mini_cluster.cc:1428] Setting key 3e09de60a60fe7ab93ca58a3220ddcb2
I20260620 14:54:56.379328 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:41289
--builtin_ntp_servers=127.5.118.148:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:54:56.381325 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.375426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:60421 (local address 127.5.118.190:41289)
0620 14:54:56.375498 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.375501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.375702 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:54:56.376325 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.376327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.376339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.376426 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:56.377181 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.377759 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.379491 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.379757 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.379869 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.380987 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:56.380989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:56.380989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:56.380995 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:56.381012 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:56.381017 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:56.381082 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.381178 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.381212 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:54:56.381536 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.375340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:41289 (local address 127.5.118.130:60421)
0620 14:54:56.375552 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:56.375598 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:56.376147 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:56.376497 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:56.376501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:56.376510 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:56.377080 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.377085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.377882 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.377883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.379372 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.379376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.379871 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.379872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.380065 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.380784 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:56.380785 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:56.380836 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:56.380874 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:56.381089 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:56.381094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:56.381097 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:54:56.381157 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:54:56.381287 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260620 14:54:56.382079 23457 heartbeater.cc:347] Connected to a master server at 127.5.118.190:41289
I20260620 14:54:56.382186 23457 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:56.382449 23457 heartbeater.cc:511] Master 127.5.118.190:41289 requested a full tablet report, sending...
I20260620 14:54:56.382920 23138 ts_manager.cc:194] Registered new tserver with Master: c687d74d791045838def0c072e5e0bd9 (127.5.118.130:33411)
I20260620 14:54:56.383358 23138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.130:60421
W20260620 14:54:56.464001 23460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:56.464254 23460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:56.464279 23460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:56.465538 23460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:56.465583 23460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:56.465610 23460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:54:56.467036 23460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:41289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:56.467326 23460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:56.467661 23460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:56.473220 23466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:56.473227 23468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:56.473227 23465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:56.473517 23460 server_base.cc:1034] running on GCE node
I20260620 14:54:56.473742 23460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:56.474030 23460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:56.475203 23460 hybrid_clock.cc:648] HybridClock initialized: now 1781967296475186 us; error 28 us; skew 500 ppm
I20260620 14:54:56.476550 23460 webserver.cc:460] Webserver started at http://127.5.118.131:40247/ using document root <none> and password file <none>
I20260620 14:54:56.476814 23460 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:56.476886 23460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:56.477034 23460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:56.478137 23460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "5aae022aa1404911b330eb42bad23a06"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "1c62a76cf39d7e724e4c623e101de288"
server_key_iv: "1a3d0b9e6a8869dab076bd22a2bc75d9"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.478518 23460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "5aae022aa1404911b330eb42bad23a06"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "1c62a76cf39d7e724e4c623e101de288"
server_key_iv: "1a3d0b9e6a8869dab076bd22a2bc75d9"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.479920 23460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:54:56.480773 23474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.480931 23460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.481035 23460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "5aae022aa1404911b330eb42bad23a06"
format_stamp: "Formatted at 2026-06-20 14:54:56 on dist-test-slave-2m5f"
server_key: "1c62a76cf39d7e724e4c623e101de288"
server_key_iv: "1a3d0b9e6a8869dab076bd22a2bc75d9"
server_key_version: "encryptionkey@0"
I20260620 14:54:56.481112 23460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:56.506083 23460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:56.506433 23460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:56.506590 23460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:56.506834 23460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:56.507174 23460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:56.507231 23460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.507272 23460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:56.507318 23460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:56.513288 23460 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:43831
I20260620 14:54:56.513376 23587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:43831 every 8 connection(s)
I20260620 14:54:56.513726 23460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:56.515522 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 23460
I20260620 14:54:56.515650 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:54:56.515898 5594 external_mini_cluster.cc:1428] Setting key 36488d46d9b75458646648143a37c8a2
I20260620 14:54:56.519912 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.514619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:51397 (local address 127.5.118.190:41289)
0620 14:54:56.514693 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.514696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.515090 (+ 394us) server_negotiation.cc:407] Connection header received
0620 14:54:56.515767 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.515768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.515785 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.515874 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:56.516793 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.517375 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.518489 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.518700 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.518777 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.519615 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:56.519617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:56.519617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:56.519623 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:56.519638 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:56.519642 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:56.519696 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.519792 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.519827 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:54:56.520256 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.514628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:41289 (local address 127.5.118.131:51397)
0620 14:54:56.514930 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:56.514968 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:56.515601 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:56.515954 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:56.515958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:56.515970 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:56.516701 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.516708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.517479 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.517480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.518385 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.518388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.518799 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.518799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.518958 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.519473 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:56.519473 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:56.519503 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:56.519526 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:56.519702 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:56.519707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:56.519711 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:54:56.519795 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:54:56.519988 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":121,"threads_started":1}
I20260620 14:54:56.520762 23588 heartbeater.cc:347] Connected to a master server at 127.5.118.190:41289
I20260620 14:54:56.520884 23588 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:56.521097 23588 heartbeater.cc:511] Master 127.5.118.190:41289 requested a full tablet report, sending...
I20260620 14:54:56.521557 23138 ts_manager.cc:194] Registered new tserver with Master: 5aae022aa1404911b330eb42bad23a06 (127.5.118.131:43831)
I20260620 14:54:56.521920 23138 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.131:51397
I20260620 14:54:56.527980 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:54:56.532928 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.529490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.5.118.190:41289)
0620 14:54:56.529589 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.529592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.529650 (+ 58us) server_negotiation.cc:407] Connection header received
0620 14:54:56.529754 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.529755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.529768 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.529832 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:56.530061 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.530827 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.531684 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.531911 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.531992 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.532670 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:56.532671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:56.532672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:56.532677 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:56.532690 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:56.532694 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:56.532742 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.532798 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.532827 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:54:56.534827 23138 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:56.536500 23138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:56.543597 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.538961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.5.118.131:43831)
0620 14:54:56.539819 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.539823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.539833 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:54:56.539869 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.539871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.539885 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.539964 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:56.540082 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.541114 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.542533 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.543029 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.543126 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.543227 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.543445 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.543490 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":817,"thread_start_us":90,"threads_started":1}
I20260620 14:54:56.543643 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.538819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:33411 (local address 127.0.0.1:36246)
0620 14:54:56.539025 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:56.539066 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:56.539138 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:56.539480 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:56.539482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:56.539484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:56.539576 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.539578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.540944 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.540945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.542324 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.542328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.543265 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.543266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.543361 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.543364 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:54:56.543415 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:54:56.543510 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":110,"thread_start_us":70,"threads_started":1}
I20260620 14:54:56.543649 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.538819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:43831 (local address 127.0.0.1:32932)
0620 14:54:56.539263 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:56.539271 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:56.539307 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:56.539981 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:56.539982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:56.539984 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:56.540018 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.540020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.541206 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.541208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.542469 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.542471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.543210 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.543211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.543306 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.543344 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:54:56.543425 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:54:56.543502 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":32,"thread_start_us":65,"threads_started":1}
I20260620 14:54:56.543831 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.538916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:37335 (local address 127.0.0.1:50940)
0620 14:54:56.539287 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:56.539299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:56.539354 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:56.539680 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:56.539681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:56.539683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:56.539732 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.539734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.540827 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.540828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.542067 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.542070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.543463 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.543464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.543515 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.543518 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:54:56.543602 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:54:56.543645 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":78,"thread_start_us":61,"threads_started":1}
I20260620 14:54:56.544065 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.539076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.5.118.130:33411)
0620 14:54:56.539293 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.539300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.539312 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:54:56.539360 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.539363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.539380 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.539468 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:56.539657 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.540854 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.542410 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.543088 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.543146 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.543227 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.543945 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.543989 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260620 14:54:56.544636 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.539117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.5.118.129:37335)
0620 14:54:56.539517 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.539521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.539530 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:54:56.539565 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.539567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.539581 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.539685 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:56.539825 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.540746 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.542183 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.543057 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.543135 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.543227 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.544499 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.544549 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":87,"threads_started":1}
I20260620 14:54:56.544617 23522 tablet_service.cc:1467] Processing CreateTablet for tablet 385b9898ebc54d4389fa854a45772579 (DEFAULT_TABLE table=test-table [id=ae0a288d5b8240189e02d6b4dc666e6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:56.544816 23391 tablet_service.cc:1467] Processing CreateTablet for tablet 385b9898ebc54d4389fa854a45772579 (DEFAULT_TABLE table=test-table [id=ae0a288d5b8240189e02d6b4dc666e6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:56.545006 23522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385b9898ebc54d4389fa854a45772579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:56.545179 23391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385b9898ebc54d4389fa854a45772579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:56.547705 23602 tablet_bootstrap.cc:492] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: Bootstrap starting.
I20260620 14:54:56.548404 23603 tablet_bootstrap.cc:492] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: Bootstrap starting.
I20260620 14:54:56.548341 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 385b9898ebc54d4389fa854a45772579 (DEFAULT_TABLE table=test-table [id=ae0a288d5b8240189e02d6b4dc666e6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:56.548583 23602 tablet_bootstrap.cc:654] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:56.548704 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385b9898ebc54d4389fa854a45772579. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:56.548933 23602 log.cc:824] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:56.549060 23603 tablet_bootstrap.cc:654] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:56.549401 23603 log.cc:824] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:56.549749 23602 tablet_bootstrap.cc:492] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: No bootstrap required, opened a new log
I20260620 14:54:56.549829 23602 ts_tablet_manager.cc:1397] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 14:54:56.550099 23603 tablet_bootstrap.cc:492] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: No bootstrap required, opened a new log
I20260620 14:54:56.550177 23603 ts_tablet_manager.cc:1397] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:54:56.551935 23603 raft_consensus.cc:348] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c30c88af6d49249c1adac5f78979c8" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 37335 } } peers { permanent_uuid: "5aae022aa1404911b330eb42bad23a06" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43831 } } peers { permanent_uuid: "c687d74d791045838def0c072e5e0bd9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33411 } }
I20260620 14:54:56.552119 23603 raft_consensus.cc:374] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:56.552165 23603 raft_consensus.cc:729] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aae022aa1404911b330eb42bad23a06, State: Initialized, Role: FOLLOWER
I20260620 14:54:56.552368 23603 consensus_queue.cc:260] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c30c88af6d49249c1adac5f78979c8" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 37335 } } peers { permanent_uuid: "5aae022aa1404911b330eb42bad23a06" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43831 } } peers { permanent_uuid: "c687d74d791045838def0c072e5e0bd9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33411 } }
I20260620 14:54:56.552492 23602 raft_consensus.cc:348] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c30c88af6d49249c1adac5f78979c8" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 37335 } } peers { permanent_uuid: "5aae022aa1404911b330eb42bad23a06" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43831 } } peers { permanent_uuid: "c687d74d791045838def0c072e5e0bd9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33411 } }
I20260620 14:54:56.552634 23603 ts_tablet_manager.cc:1428] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:54:56.552626 23606 tablet_bootstrap.cc:492] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8: Bootstrap starting.
I20260620 14:54:56.552672 23602 raft_consensus.cc:374] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:56.552714 23602 raft_consensus.cc:729] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c687d74d791045838def0c072e5e0bd9, State: Initialized, Role: FOLLOWER
I20260620 14:54:56.552748 23588 heartbeater.cc:503] Master 127.5.118.190:41289 was elected leader, sending a full tablet report...
I20260620 14:54:56.552848 23602 consensus_queue.cc:260] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c30c88af6d49249c1adac5f78979c8" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 37335 } } peers { permanent_uuid: "5aae022aa1404911b330eb42bad23a06" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43831 } } peers { permanent_uuid: "c687d74d791045838def0c072e5e0bd9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33411 } }
I20260620 14:54:56.553107 23602 ts_tablet_manager.cc:1428] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 14:54:56.553220 23457 heartbeater.cc:503] Master 127.5.118.190:41289 was elected leader, sending a full tablet report...
I20260620 14:54:56.553310 23606 tablet_bootstrap.cc:654] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:56.553632 23606 log.cc:824] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:56.554402 23606 tablet_bootstrap.cc:492] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8: No bootstrap required, opened a new log
I20260620 14:54:56.554456 23606 ts_tablet_manager.cc:1397] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:54:56.556401 23606 raft_consensus.cc:348] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c30c88af6d49249c1adac5f78979c8" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 37335 } } peers { permanent_uuid: "5aae022aa1404911b330eb42bad23a06" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43831 } } peers { permanent_uuid: "c687d74d791045838def0c072e5e0bd9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33411 } }
I20260620 14:54:56.556526 23606 raft_consensus.cc:374] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:56.556547 23606 raft_consensus.cc:729] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c30c88af6d49249c1adac5f78979c8, State: Initialized, Role: FOLLOWER
I20260620 14:54:56.556656 23606 consensus_queue.cc:260] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c30c88af6d49249c1adac5f78979c8" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 37335 } } peers { permanent_uuid: "5aae022aa1404911b330eb42bad23a06" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43831 } } peers { permanent_uuid: "c687d74d791045838def0c072e5e0bd9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33411 } }
I20260620 14:54:56.556932 23606 ts_tablet_manager.cc:1428] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:54:56.556979 23326 heartbeater.cc:503] Master 127.5.118.190:41289 was elected leader, sending a full tablet report...
W20260620 14:54:56.625131 23458 tablet.cc:2369] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:54:56.733498 23327 tablet.cc:2369] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:54:56.738551 23608 raft_consensus.cc:482] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:56.738691 23608 raft_consensus.cc:504] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c30c88af6d49249c1adac5f78979c8" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 37335 } } peers { permanent_uuid: "5aae022aa1404911b330eb42bad23a06" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43831 } } peers { permanent_uuid: "c687d74d791045838def0c072e5e0bd9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33411 } }
I20260620 14:54:56.739190 23608 leader_election.cc:288] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03c30c88af6d49249c1adac5f78979c8 (127.5.118.129:37335), 5aae022aa1404911b330eb42bad23a06 (127.5.118.131:43831)
I20260620 14:54:56.743578 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.739526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:49253 (local address 127.5.118.129:37335)
0620 14:54:56.739603 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.739605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.739617 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:54:56.739649 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.739650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.739663 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.739763 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:56.740025 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.740872 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.742530 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.743135 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.743216 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.743224 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.743465 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.743505 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:54:56.743578 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.739541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:38803 (local address 127.5.118.131:43831)
0620 14:54:56.739586 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.739589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.740026 (+ 437us) server_negotiation.cc:407] Connection header received
0620 14:54:56.740063 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.740065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.740076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.740145 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:56.740351 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.741069 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.742542 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.743135 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.743216 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.743224 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.743465 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.743503 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:54:56.743631 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.739438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:37335 (local address 127.5.118.130:49253)
0620 14:54:56.739489 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:56.739502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:56.739575 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:56.739771 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:56.739788 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:56.739791 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:56.739875 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.739877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.740971 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.740972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.742399 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.742403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.743262 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.743294 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.743364 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.743371 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:54:56.743441 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:54:56.743497 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":6}
I20260620 14:54:56.743932 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.739445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:43831 (local address 127.5.118.130:38803)
0620 14:54:56.739852 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:56.739865 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:56.740005 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:56.740148 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:56.740148 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:56.740150 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:56.740270 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.740272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.741144 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.741145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.742452 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:56.742455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.743265 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.743266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:56.743364 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.743371 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:54:56.743441 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 14:54:56.743494 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":196,"thread_start_us":140,"threads_started":1}
I20260620 14:54:56.744040 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b9898ebc54d4389fa854a45772579" candidate_uuid: "c687d74d791045838def0c072e5e0bd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03c30c88af6d49249c1adac5f78979c8" is_pre_election: true
I20260620 14:54:56.744264 23280 raft_consensus.cc:2454] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c687d74d791045838def0c072e5e0bd9 in term 0.
I20260620 14:54:56.744422 23542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b9898ebc54d4389fa854a45772579" candidate_uuid: "c687d74d791045838def0c072e5e0bd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aae022aa1404911b330eb42bad23a06" is_pre_election: true
I20260620 14:54:56.744580 23542 raft_consensus.cc:2454] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c687d74d791045838def0c072e5e0bd9 in term 0.
I20260620 14:54:56.744572 23346 leader_election.cc:302] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03c30c88af6d49249c1adac5f78979c8, c687d74d791045838def0c072e5e0bd9; no voters:
I20260620 14:54:56.744710 23608 raft_consensus.cc:2781] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:56.744802 23608 raft_consensus.cc:482] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:56.744843 23608 raft_consensus.cc:3037] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:56.745551 23608 raft_consensus.cc:504] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c30c88af6d49249c1adac5f78979c8" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 37335 } } peers { permanent_uuid: "5aae022aa1404911b330eb42bad23a06" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43831 } } peers { permanent_uuid: "c687d74d791045838def0c072e5e0bd9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33411 } }
I20260620 14:54:56.745744 23608 leader_election.cc:288] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [CANDIDATE]: Term 1 election: Requested vote from peers 03c30c88af6d49249c1adac5f78979c8 (127.5.118.129:37335), 5aae022aa1404911b330eb42bad23a06 (127.5.118.131:43831)
I20260620 14:54:56.745906 23542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b9898ebc54d4389fa854a45772579" candidate_uuid: "c687d74d791045838def0c072e5e0bd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aae022aa1404911b330eb42bad23a06"
I20260620 14:54:56.746025 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b9898ebc54d4389fa854a45772579" candidate_uuid: "c687d74d791045838def0c072e5e0bd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03c30c88af6d49249c1adac5f78979c8"
I20260620 14:54:56.745998 23542 raft_consensus.cc:3037] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:56.746114 23280 raft_consensus.cc:3037] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:56.746896 23542 raft_consensus.cc:2454] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c687d74d791045838def0c072e5e0bd9 in term 1.
I20260620 14:54:56.747035 23280 raft_consensus.cc:2454] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c687d74d791045838def0c072e5e0bd9 in term 1.
I20260620 14:54:56.747080 23345 leader_election.cc:302] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aae022aa1404911b330eb42bad23a06, c687d74d791045838def0c072e5e0bd9; no voters:
I20260620 14:54:56.747195 23608 raft_consensus.cc:2781] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:56.747365 23608 raft_consensus.cc:686] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 1 LEADER]: Becoming Leader. State: Replica: c687d74d791045838def0c072e5e0bd9, State: Running, Role: LEADER
I20260620 14:54:56.747486 23608 consensus_queue.cc:237] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c30c88af6d49249c1adac5f78979c8" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 37335 } } peers { permanent_uuid: "5aae022aa1404911b330eb42bad23a06" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43831 } } peers { permanent_uuid: "c687d74d791045838def0c072e5e0bd9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33411 } }
I20260620 14:54:56.748435 23137 catalog_manager.cc:5495] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c687d74d791045838def0c072e5e0bd9 (127.5.118.130). New cstate: current_term: 1 leader_uuid: "c687d74d791045838def0c072e5e0bd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03c30c88af6d49249c1adac5f78979c8" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 37335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aae022aa1404911b330eb42bad23a06" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c687d74d791045838def0c072e5e0bd9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 33411 } health_report { overall_health: HEALTHY } } }
W20260620 14:54:56.764328 23589 tablet.cc:2369] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:54:56.780351 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.777625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.5.118.130:33411)
0620 14:54:56.777708 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:56.777710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:56.777727 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:54:56.777828 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:56.777829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:56.777850 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:56.777919 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:56.778061 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.778540 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.779422 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:56.779638 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:56.779700 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:56.779953 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:56.780060 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:56.780129 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:56.780235 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 14:54:56.780263 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:54:56.782371 23542 raft_consensus.cc:1261] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06 [term 1 FOLLOWER]: Refusing update from remote peer c687d74d791045838def0c072e5e0bd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:56.782363 23280 raft_consensus.cc:1261] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8 [term 1 FOLLOWER]: Refusing update from remote peer c687d74d791045838def0c072e5e0bd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:56.782862 23612 consensus_queue.cc:1035] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aae022aa1404911b330eb42bad23a06" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 43831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:56.783057 23608 consensus_queue.cc:1035] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03c30c88af6d49249c1adac5f78979c8" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:56.785917 23616 mvcc.cc:205] Tried to move back new op lower bound from 7298938047617134592 to 7298938047478259712. Current Snapshot: MvccSnapshot[applied={T|T < 7298938047617134592}]
I20260620 14:54:56.786219 23617 mvcc.cc:205] Tried to move back new op lower bound from 7298938047617134592 to 7298938047478259712. Current Snapshot: MvccSnapshot[applied={T|T < 7298938047617134592}]
I20260620 14:54:56.788445 23138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:56.788559 23138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:56.789315 23138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 599b5b3d64b040cb8ce48eab2a00f5f4: Sending DeleteTablet for 3 replicas of tablet 385b9898ebc54d4389fa854a45772579
I20260620 14:54:56.789640 23260 tablet_service.cc:1514] Processing DeleteTablet for tablet 385b9898ebc54d4389fa854a45772579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:56 UTC) from {username='slave'} at 127.0.0.1:50940
I20260620 14:54:56.789675 23522 tablet_service.cc:1514] Processing DeleteTablet for tablet 385b9898ebc54d4389fa854a45772579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:56 UTC) from {username='slave'} at 127.0.0.1:32932
I20260620 14:54:56.789799 23391 tablet_service.cc:1514] Processing DeleteTablet for tablet 385b9898ebc54d4389fa854a45772579 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:56 UTC) from {username='slave'} at 127.0.0.1:36246
I20260620 14:54:56.789934 23626 tablet_replica.cc:330] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: stopping tablet replica
I20260620 14:54:56.790046 23626 raft_consensus.cc:2229] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:56.790169 23626 raft_consensus.cc:2258] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:56.790340 23625 tablet_replica.cc:330] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8: stopping tablet replica
I20260620 14:54:56.790455 23625 raft_consensus.cc:2229] T 385b9898ebc54d4389fa854a45772579 P 03c30c88af6d49249c1adac5f78979c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:56.790463 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 23198
I20260620 14:54:56.790786 23626 ts_tablet_manager.cc:1905] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:56.791275 23627 tablet_replica.cc:330] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: stopping tablet replica
I20260620 14:54:56.791386 23627 raft_consensus.cc:2229] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:56.791523 23627 raft_consensus.cc:2258] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:56.792414 23627 ts_tablet_manager.cc:1905] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:56.792775 23626 ts_tablet_manager.cc:1918] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:56.792912 23626 log.cc:1196] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/385b9898ebc54d4389fa854a45772579
I20260620 14:54:56.793239 23626 ts_tablet_manager.cc:1939] T 385b9898ebc54d4389fa854a45772579 P 5aae022aa1404911b330eb42bad23a06: Deleting consensus metadata
I20260620 14:54:56.793810 23123 catalog_manager.cc:4841] TS 5aae022aa1404911b330eb42bad23a06 (127.5.118.131:43831): tablet 385b9898ebc54d4389fa854a45772579 (table test-table [id=ae0a288d5b8240189e02d6b4dc666e6b]) successfully deleted
I20260620 14:54:56.794483 23627 ts_tablet_manager.cc:1918] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:56.794565 23627 log.cc:1196] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/385b9898ebc54d4389fa854a45772579
I20260620 14:54:56.794857 23627 ts_tablet_manager.cc:1939] T 385b9898ebc54d4389fa854a45772579 P c687d74d791045838def0c072e5e0bd9: Deleting consensus metadata
I20260620 14:54:56.795300 23125 catalog_manager.cc:4841] TS c687d74d791045838def0c072e5e0bd9 (127.5.118.130:33411): tablet 385b9898ebc54d4389fa854a45772579 (table test-table [id=ae0a288d5b8240189e02d6b4dc666e6b]) successfully deleted
W20260620 14:54:56.797998 23124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:37335 (error 108)
I20260620 14:54:56.798597 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:56.798442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:37335 (local address 127.0.0.1:50944)
0620 14:54:56.798508 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:56.798559 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:37335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:54:56.798677 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 23329
W20260620 14:54:56.799888 23124 catalog_manager.cc:4584] TS 03c30c88af6d49249c1adac5f78979c8 (127.5.118.129:37335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 385b9898ebc54d4389fa854a45772579: Network error: Client connection negotiation failed: client connection to 127.5.118.129:37335: connect: Connection refused (error 111)
I20260620 14:54:56.804436 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 23460
I20260620 14:54:56.809391 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 23107
2026-06-20T14:54:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (923 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:54:56 dist-test-slave-2m5f krb5kdc[23631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:54:56 dist-test-slave-2m5f krb5kdc[23631](info): set up 2 sockets
Jun 20 14:54:56 dist-test-slave-2m5f krb5kdc[23631](info): commencing operation
krb5kdc: starting...
W20260620 14:54:57.928119 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:54:58 dist-test-slave-2m5f krb5kdc[23631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:54:58.052137 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:37095
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:37095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:58.138669 23644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:58.138890 23644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:58.138916 23644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:58.140125 23644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:58.140163 23644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:58.140250 23644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:58.140275 23644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:58.140302 23644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:58.141868 23644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:37095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:37095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:58.142089 23644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:58.142277 23644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:58.147270 23652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:58.147270 23650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:58.147284 23649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:58.147382 23644 server_base.cc:1034] running on GCE node
I20260620 14:54:58.147858 23644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:58.148447 23644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:58.149673 23644 hybrid_clock.cc:648] HybridClock initialized: now 1781967298149641 us; error 44 us; skew 500 ppm
Jun 20 14:54:58 dist-test-slave-2m5f krb5kdc[23631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:58.153118 23644 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:54:58.153627 23644 webserver.cc:460] Webserver started at http://127.5.118.190:41911/ using document root <none> and password file <none>
I20260620 14:54:58.153882 23644 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:58.153959 23644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:58.154109 23644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:58.155318 23644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "18ed9427a89a4f25b7f2ad2334babadb"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "99d363247185eedf0af399188e45812f"
server_key_iv: "4329faccc2e51491b9a15d66d4c14ae9"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.155738 23644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "18ed9427a89a4f25b7f2ad2334babadb"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "99d363247185eedf0af399188e45812f"
server_key_iv: "4329faccc2e51491b9a15d66d4c14ae9"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.157259 23644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:54:58.158077 23659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.158342 23644 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:58.158453 23644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "18ed9427a89a4f25b7f2ad2334babadb"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "99d363247185eedf0af399188e45812f"
server_key_iv: "4329faccc2e51491b9a15d66d4c14ae9"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.158535 23644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:58.177847 23644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:58.180701 23644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:58.180884 23644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:58.185073 23644 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:37095
I20260620 14:54:58.185110 23711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:37095 every 8 connection(s)
I20260620 14:54:58.185483 23644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:58.186424 23712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:58.187614 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 23644
I20260620 14:54:58.187755 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:54:58.188002 5594 external_mini_cluster.cc:1428] Setting key b3f9490e5bafc4f520d9b332a46fab05
I20260620 14:54:58.189378 23712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb: Bootstrap starting.
I20260620 14:54:58.190121 23712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:58.190480 23712 log.cc:824] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:58.191219 23712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb: No bootstrap required, opened a new log
I20260620 14:54:58.193624 23712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ed9427a89a4f25b7f2ad2334babadb" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37095 } }
I20260620 14:54:58.193795 23712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:58.193833 23712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ed9427a89a4f25b7f2ad2334babadb, State: Initialized, Role: FOLLOWER
I20260620 14:54:58.193989 23712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ed9427a89a4f25b7f2ad2334babadb" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37095 } }
I20260620 14:54:58.194085 23712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:58.194123 23712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:58.194171 23712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:58.194980 23712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ed9427a89a4f25b7f2ad2334babadb" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37095 } }
I20260620 14:54:58.195104 23712 leader_election.cc:302] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18ed9427a89a4f25b7f2ad2334babadb; no voters:
I20260620 14:54:58.195323 23712 leader_election.cc:288] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:58.195348 23717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:58.195472 23717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [term 1 LEADER]: Becoming Leader. State: Replica: 18ed9427a89a4f25b7f2ad2334babadb, State: Running, Role: LEADER
I20260620 14:54:58.195547 23717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ed9427a89a4f25b7f2ad2334babadb" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37095 } }
I20260620 14:54:58.195613 23712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 14:54:58 dist-test-slave-2m5f krb5kdc[23631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:54:58.196403 23718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18ed9427a89a4f25b7f2ad2334babadb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ed9427a89a4f25b7f2ad2334babadb" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37095 } } }
I20260620 14:54:58.196497 23718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:58.196851 23719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18ed9427a89a4f25b7f2ad2334babadb. Latest consensus state: current_term: 1 leader_uuid: "18ed9427a89a4f25b7f2ad2334babadb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ed9427a89a4f25b7f2ad2334babadb" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37095 } } }
I20260620 14:54:58.196925 23719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:58.197280 23726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:58.197999 23726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:58.199599 23726 catalog_manager.cc:1347] Generated new cluster ID: 7d55ed2a86db49a886e8c9f0968e5e85
I20260620 14:54:58.199656 23726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:58.200788 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.188594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.5.118.190:37095)
0620 14:54:58.188795 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:58.188807 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:58.188845 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:54:58.189373 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:58.189392 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:58.189458 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:58.189606 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:58.190596 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.191647 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.192791 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.193206 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.193294 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.196332 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:58.196337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:58.196337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:58.196353 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:58.198913 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.199406 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.199409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.199409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.199539 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.199774 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.199775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.199776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.200104 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:58.200271 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:58.200437 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:54:58.200584 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20260620 14:54:58.227752 23726 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:58.228564 23726 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:58.242955 23726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb: Generated new TSK 0
I20260620 14:54:58.243237 23726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:54:58.283978 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:37095
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:58.368834 23738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:58.369084 23738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:58.369125 23738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:58.370344 23738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:58.370429 23738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:58.370477 23738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:54:58.372036 23738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:58.372362 23738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:58.372632 23738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:58.378243 23743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:58.378253 23746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:58.378306 23738 server_base.cc:1034] running on GCE node
W20260620 14:54:58.378253 23744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:58.378767 23738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:58.379114 23738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:58.380237 23738 hybrid_clock.cc:648] HybridClock initialized: now 1781967298380224 us; error 32 us; skew 500 ppm
Jun 20 14:54:58 dist-test-slave-2m5f krb5kdc[23631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:58.382817 23738 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:54:58.383227 23738 webserver.cc:460] Webserver started at http://127.5.118.129:41765/ using document root <none> and password file <none>
I20260620 14:54:58.383414 23738 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:58.383463 23738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:58.383553 23738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:58.384723 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "9de7cd99b605487e873fb11c786a0a5b"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "e70e253f74b451835c90b867fd6867ac"
server_key_iv: "082214cbfc6e5521c194f37e1b3a238f"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.385030 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "9de7cd99b605487e873fb11c786a0a5b"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "e70e253f74b451835c90b867fd6867ac"
server_key_iv: "082214cbfc6e5521c194f37e1b3a238f"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.386330 23738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:54:58.387079 23753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.387238 23738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:54:58.387331 23738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "9de7cd99b605487e873fb11c786a0a5b"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "e70e253f74b451835c90b867fd6867ac"
server_key_iv: "082214cbfc6e5521c194f37e1b3a238f"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.387408 23738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:58.399672 23738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:58.402140 23738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:58.402267 23738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:58.402485 23738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:58.402822 23738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:58.402860 23738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.402884 23738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:58.402897 23738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.408730 23738 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:43475
I20260620 14:54:58.408807 23866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:43475 every 8 connection(s)
I20260620 14:54:58.409145 23738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:58.409667 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 23738
I20260620 14:54:58.409793 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:58.410049 5594 external_mini_cluster.cc:1428] Setting key cd240f155e9e7ba976ba924dd7424d86
Jun 20 14:54:58 dist-test-slave-2m5f krb5kdc[23631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
I20260620 14:54:58.420326 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.410273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:41533 (local address 127.5.118.190:37095)
0620 14:54:58.410351 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:58.410354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:58.410363 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:54:58.410990 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:58.410991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:58.411005 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:58.411078 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:58.412107 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.412677 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.413851 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.414218 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.414305 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.416843 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:58.416845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:58.416846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:58.416855 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:58.418582 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.419095 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.419096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.419096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.419149 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.419454 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.419455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.419455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.419568 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:58.419625 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:58.420099 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 14:54:58.420237 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:54:58.420545 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.409976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:37095 (local address 127.5.118.129:41533)
0620 14:54:58.410175 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:58.410247 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:58.410861 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:58.411134 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:58.411137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:58.411706 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:58.411976 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.411985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.412773 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.412774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.413753 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.413755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.414310 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.414310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.414422 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.415010 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:58.415019 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:58.416727 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:58.418716 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:58.418727 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:58.418730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:58.419011 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:58.419239 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:58.419240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:58.419240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:58.419386 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:58.419672 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:58.419680 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:58.419943 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 14:54:58.420069 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:54:58.420275 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260620 14:54:58.421160 23867 heartbeater.cc:347] Connected to a master server at 127.5.118.190:37095
I20260620 14:54:58.421253 23867 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:58.421408 23867 heartbeater.cc:511] Master 127.5.118.190:37095 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
I20260620 14:54:58.422047 23676 ts_manager.cc:194] Registered new tserver with Master: 9de7cd99b605487e873fb11c786a0a5b (127.5.118.129:43475)
I20260620 14:54:58.422969 23676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:41533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:54:58.438737 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:37095
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:58.521610 23872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:58.521878 23872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:58.521917 23872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:58.523116 23872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:58.523188 23872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:58.523236 23872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:54:58.524775 23872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:58.525058 23872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:58.525301 23872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:58.530496 23880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:58.530458 23878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:58.530505 23877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:58.530614 23872 server_base.cc:1034] running on GCE node
I20260620 14:54:58.530923 23872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:58.531206 23872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:58.532382 23872 hybrid_clock.cc:648] HybridClock initialized: now 1781967298532363 us; error 46 us; skew 500 ppm
Jun 20 14:54:58 dist-test-slave-2m5f krb5kdc[23631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:58.535028 23872 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:54:58.535524 23872 webserver.cc:460] Webserver started at http://127.5.118.130:37519/ using document root <none> and password file <none>
I20260620 14:54:58.535765 23872 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:58.535837 23872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:58.536005 23872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:58.537177 23872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "fbf69881f4554b86acee25260d8e0151"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "9b43c75075f69f98e945f68bac0cf4c9"
server_key_iv: "3d60dba61a140e117487cff63597082d"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.537580 23872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "fbf69881f4554b86acee25260d8e0151"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "9b43c75075f69f98e945f68bac0cf4c9"
server_key_iv: "3d60dba61a140e117487cff63597082d"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.538995 23872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:54:58.539760 23887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.539958 23872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:54:58.540064 23872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "fbf69881f4554b86acee25260d8e0151"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "9b43c75075f69f98e945f68bac0cf4c9"
server_key_iv: "3d60dba61a140e117487cff63597082d"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.540150 23872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:58.555845 23872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:58.558374 23872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:58.558573 23872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:58.558817 23872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:58.559151 23872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:58.559208 23872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.559253 23872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:58.559278 23872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.565303 23872 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:40959
I20260620 14:54:58.565346 24000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:40959 every 8 connection(s)
I20260620 14:54:58.565676 23872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:54:58 dist-test-slave-2m5f krb5kdc[23631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:54:58.574040 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 23872
I20260620 14:54:58.574183 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:58.574479 5594 external_mini_cluster.cc:1428] Setting key b169ed7a5fdcb5b2c36fdca18626dee3
I20260620 14:54:58.575695 24003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.566376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:37095 (local address 127.5.118.130:47283)
0620 14:54:58.566547 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:58.566591 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:58.567076 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:58.567288 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:58.567291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:58.567887 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:58.568122 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.568128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.568912 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.568913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.569740 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.569742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.570170 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.570171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.570303 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.570868 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:58.570877 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:58.572424 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:58.573918 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:58.573927 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:58.573929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:58.574258 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:58.574526 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:58.574527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:58.574527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:58.574675 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:58.575057 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:58.575064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:58.575226 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 14:54:58.575348 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:54:58.575526 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260620 14:54:58.575722 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.566456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:47283 (local address 127.5.118.190:37095)
0620 14:54:58.566503 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:58.566505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:58.566682 (+ 177us) server_negotiation.cc:407] Connection header received
0620 14:54:58.567155 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:58.567156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:58.567165 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:58.567231 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:58.568277 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.568798 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.569854 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.570068 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.570158 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.572540 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:58.572542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:58.572543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:58.572551 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:58.573824 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.574364 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.574366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.574367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.574450 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.574779 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.574781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.574781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.574974 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:58.575050 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:58.575513 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 14:54:58.575609 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:54:58.577172 24001 heartbeater.cc:347] Connected to a master server at 127.5.118.190:37095
I20260620 14:54:58.577332 24001 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:58.577545 24001 heartbeater.cc:511] Master 127.5.118.190:37095 requested a full tablet report, sending...
I20260620 14:54:58.578001 23676 ts_manager.cc:194] Registered new tserver with Master: fbf69881f4554b86acee25260d8e0151 (127.5.118.130:40959)
I20260620 14:54:58.578541 23676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:47283
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:54:58.602905 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:37095
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:44383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:54:58.686707 24006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:58.686933 24006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:58.686959 24006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:58.688263 24006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:58.688308 24006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:58.688336 24006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:54:58.689859 24006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:44383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:37095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:58.690068 24006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:58.690315 24006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:58.695581 24011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:58.695600 24012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:58.695576 24014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:58.695737 24006 server_base.cc:1034] running on GCE node
I20260620 14:54:58.696008 24006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:58.696298 24006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:58.697502 24006 hybrid_clock.cc:648] HybridClock initialized: now 1781967298697484 us; error 34 us; skew 500 ppm
Jun 20 14:54:58 dist-test-slave-2m5f krb5kdc[23631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:54:58.700076 24006 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:54:58.700577 24006 webserver.cc:460] Webserver started at http://127.5.118.131:36239/ using document root <none> and password file <none>
I20260620 14:54:58.700912 24006 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:58.700990 24006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:58.701140 24006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:58.702222 24006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "844b8c55295f4b36b37c853661900d77"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "c85b134a01460c70a7082776ca7ba3c8"
server_key_iv: "49a8578d3e4024719b0e4806b9423ef3"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.702543 24006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "844b8c55295f4b36b37c853661900d77"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "c85b134a01460c70a7082776ca7ba3c8"
server_key_iv: "49a8578d3e4024719b0e4806b9423ef3"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.703848 24006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:54:58.704824 24021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.705052 24006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:54:58.705144 24006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "844b8c55295f4b36b37c853661900d77"
format_stamp: "Formatted at 2026-06-20 14:54:58 on dist-test-slave-2m5f"
server_key: "c85b134a01460c70a7082776ca7ba3c8"
server_key_iv: "49a8578d3e4024719b0e4806b9423ef3"
server_key_version: "encryptionkey@0"
I20260620 14:54:58.705209 24006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:58.720472 24006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:58.722605 24006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:58.722721 24006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:58.722915 24006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:58.723205 24006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:58.723238 24006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.723260 24006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:58.723273 24006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:58.729288 24006 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:34893
I20260620 14:54:58.729389 24134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:34893 every 8 connection(s)
I20260620 14:54:58.729717 24006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:54:58 dist-test-slave-2m5f krb5kdc[23631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967298, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:54:58.738349 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 24006
I20260620 14:54:58.738476 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:54:58.738687 5594 external_mini_cluster.cc:1428] Setting key e27139602b6c265a8d220d5ce05189e2
I20260620 14:54:58.740958 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.730525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:55747 (local address 127.5.118.190:37095)
0620 14:54:58.730635 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:58.730637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:58.730777 (+ 140us) server_negotiation.cc:407] Connection header received
0620 14:54:58.731387 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:58.731388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:58.731400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:58.731471 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:58.732573 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.733071 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.734225 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.734425 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.734507 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.737264 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:58.737266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:58.737266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:58.737274 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:58.739340 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.739812 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.739814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.739814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.739886 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.740203 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.740204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.740204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.740346 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:58.740438 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:58.740753 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:54:58.740854 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 14:54:58.741194 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.730449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:37095 (local address 127.5.118.131:55747)
0620 14:54:58.730640 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:58.730683 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:58.731272 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:58.731533 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:58.731537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:58.732241 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:58.732481 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.732488 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.733161 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.733161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.734105 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.734108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.734533 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.734534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.734659 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.735318 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:54:58.735328 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:58.737146 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:58.739414 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:58.739425 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:58.739428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:58.739713 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:58.739967 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:54:58.739968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:54:58.739969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:54:58.740098 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:54:58.740423 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:58.740444 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:58.740598 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 14:54:58.740715 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:54:58.740910 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260620 14:54:58.741786 24135 heartbeater.cc:347] Connected to a master server at 127.5.118.190:37095
I20260620 14:54:58.741907 24135 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:58.742089 24135 heartbeater.cc:511] Master 127.5.118.190:37095 requested a full tablet report, sending...
I20260620 14:54:58.742522 23676 ts_manager.cc:194] Registered new tserver with Master: 844b8c55295f4b36b37c853661900d77 (127.5.118.131:34893)
I20260620 14:54:58.742942 23676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:55747
I20260620 14:54:58.751180 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:54:58.759119 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.752715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.5.118.190:37095)
0620 14:54:58.752776 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:58.752784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:58.752927 (+ 143us) server_negotiation.cc:407] Connection header received
0620 14:54:58.752975 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:58.752976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:58.752985 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:58.753041 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:58.753588 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.754064 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.754896 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.755079 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.755129 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.756658 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:58.756659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:58.756659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:54:58.756665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:58.757981 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.758372 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.758373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.758373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.758427 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:54:58.758603 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:54:58.758603 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:54:58.758603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:54:58.758740 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:58.758886 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:58.759003 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 14:54:58.759054 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:54:58.761090 23676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:58.762820 23676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:58.769769 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.765306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:34893 (local address 127.0.0.1:42670)
0620 14:54:58.765757 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:58.765768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:58.765810 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:58.766258 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:58.766258 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:58.766260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:58.766305 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.766306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.767586 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.767587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.768675 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.768678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.769531 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.769532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.769592 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.769605 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:54:58.769661 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:54:58.769704 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":5,"thread_start_us":84,"threads_started":1}
I20260620 14:54:58.769834 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.765747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.5.118.131:34893)
0620 14:54:58.765948 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:58.765952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:58.765963 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:54:58.766136 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:58.766138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:58.766157 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:58.766372 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:58.766453 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.767508 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.768769 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.769431 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.769520 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.769581 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:58.769685 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 14:54:58.769730 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260620 14:54:58.770247 24146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.765306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:40959 (local address 127.0.0.1:58520)
0620 14:54:58.765514 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:58.765559 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:58.765635 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:58.766029 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:58.766033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:58.766035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:58.766123 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.766125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.767534 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.767535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.768955 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.768957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.769831 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.769832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.770072 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.770077 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:54:58.770131 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 14:54:58.770173 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":36,"thread_start_us":68,"threads_started":1}
I20260620 14:54:58.770574 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.765536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.5.118.130:40959)
0620 14:54:58.765728 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:58.765734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:58.765745 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:54:58.765793 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:58.765795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:58.765808 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:58.765910 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:58.766255 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.767433 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.769045 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.769741 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.769812 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.769869 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:58.770445 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 14:54:58.770493 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":96,"threads_started":1}
I20260620 14:54:58.770769 24069 tablet_service.cc:1467] Processing CreateTablet for tablet bedbd83c745440c0beb79544d3082249 (DEFAULT_TABLE table=test-table [id=8881ff0f3f5b4d4f9b4e8249ce6c595a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:58.771157 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.765312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:43475 (local address 127.0.0.1:37550)
0620 14:54:58.765887 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:58.765952 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:58.765999 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:58.766301 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:58.766302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:58.766305 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:58.766355 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.766357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.767326 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.767327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.768953 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:58.768957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.770900 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.770902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:58.770997 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.771002 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:54:58.771054 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:54:58.771097 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":100,"thread_start_us":103,"threads_started":1}
I20260620 14:54:58.771181 24069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bedbd83c745440c0beb79544d3082249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:58.771239 23935 tablet_service.cc:1467] Processing CreateTablet for tablet bedbd83c745440c0beb79544d3082249 (DEFAULT_TABLE table=test-table [id=8881ff0f3f5b4d4f9b4e8249ce6c595a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:58.771482 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:58.765563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.5.118.129:43475)
0620 14:54:58.766000 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:58.766006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:58.766037 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:54:58.766078 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:58.766083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:58.766125 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:58.766206 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:58.766441 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.767220 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.769919 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:58.770753 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:58.770814 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:58.770867 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:58.771349 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 14:54:58.771399 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":105,"threads_started":1}
I20260620 14:54:58.771581 23935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bedbd83c745440c0beb79544d3082249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:58.772060 23801 tablet_service.cc:1467] Processing CreateTablet for tablet bedbd83c745440c0beb79544d3082249 (DEFAULT_TABLE table=test-table [id=8881ff0f3f5b4d4f9b4e8249ce6c595a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:58.772406 23801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bedbd83c745440c0beb79544d3082249. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:58.775008 24153 tablet_bootstrap.cc:492] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: Bootstrap starting.
I20260620 14:54:58.775239 24152 tablet_bootstrap.cc:492] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: Bootstrap starting.
I20260620 14:54:58.775455 24154 tablet_bootstrap.cc:492] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b: Bootstrap starting.
I20260620 14:54:58.775997 24152 tablet_bootstrap.cc:654] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:58.776063 24154 tablet_bootstrap.cc:654] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:58.776671 24152 log.cc:824] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:58.776671 24154 log.cc:824] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:58.776796 24153 tablet_bootstrap.cc:654] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:58.777384 24153 log.cc:824] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:58.777547 24154 tablet_bootstrap.cc:492] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b: No bootstrap required, opened a new log
I20260620 14:54:58.777611 24152 tablet_bootstrap.cc:492] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: No bootstrap required, opened a new log
I20260620 14:54:58.777635 24154 ts_tablet_manager.cc:1397] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:54:58.777683 24152 ts_tablet_manager.cc:1397] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:54:58.779294 24152 raft_consensus.cc:348] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 } } peers { permanent_uuid: "844b8c55295f4b36b37c853661900d77" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 34893 } } peers { permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 } }
I20260620 14:54:58.779446 24152 raft_consensus.cc:374] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:58.779469 24152 raft_consensus.cc:729] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 844b8c55295f4b36b37c853661900d77, State: Initialized, Role: FOLLOWER
I20260620 14:54:58.779419 24154 raft_consensus.cc:348] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 } } peers { permanent_uuid: "844b8c55295f4b36b37c853661900d77" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 34893 } } peers { permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 } }
I20260620 14:54:58.779603 24152 consensus_queue.cc:260] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 } } peers { permanent_uuid: "844b8c55295f4b36b37c853661900d77" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 34893 } } peers { permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 } }
I20260620 14:54:58.779665 24154 raft_consensus.cc:374] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:58.779722 24154 raft_consensus.cc:729] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de7cd99b605487e873fb11c786a0a5b, State: Initialized, Role: FOLLOWER
I20260620 14:54:58.779887 24152 ts_tablet_manager.cc:1428] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:54:58.779858 24154 consensus_queue.cc:260] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 } } peers { permanent_uuid: "844b8c55295f4b36b37c853661900d77" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 34893 } } peers { permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 } }
I20260620 14:54:58.780133 24154 ts_tablet_manager.cc:1428] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:54:58.780234 23867 heartbeater.cc:503] Master 127.5.118.190:37095 was elected leader, sending a full tablet report...
I20260620 14:54:58.780297 24135 heartbeater.cc:503] Master 127.5.118.190:37095 was elected leader, sending a full tablet report...
I20260620 14:54:58.781318 24153 tablet_bootstrap.cc:492] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: No bootstrap required, opened a new log
I20260620 14:54:58.781404 24153 ts_tablet_manager.cc:1397] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 14:54:58.782871 24153 raft_consensus.cc:348] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 } } peers { permanent_uuid: "844b8c55295f4b36b37c853661900d77" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 34893 } } peers { permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 } }
I20260620 14:54:58.783063 24153 raft_consensus.cc:374] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:58.783093 24153 raft_consensus.cc:729] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbf69881f4554b86acee25260d8e0151, State: Initialized, Role: FOLLOWER
I20260620 14:54:58.783212 24153 consensus_queue.cc:260] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 } } peers { permanent_uuid: "844b8c55295f4b36b37c853661900d77" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 34893 } } peers { permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 } }
I20260620 14:54:58.783535 24153 ts_tablet_manager.cc:1428] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:54:58.783593 24001 heartbeater.cc:503] Master 127.5.118.190:37095 was elected leader, sending a full tablet report...
W20260620 14:54:58.816272 24002 tablet.cc:2369] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:54:58.909883 23868 tablet.cc:2369] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:54:58.980392 24136 tablet.cc:2369] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:54:59.167879 24158 raft_consensus.cc:482] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:59.168025 24158 raft_consensus.cc:504] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 } } peers { permanent_uuid: "844b8c55295f4b36b37c853661900d77" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 34893 } } peers { permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 } }
I20260620 14:54:59.168527 24158 leader_election.cc:288] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbf69881f4554b86acee25260d8e0151 (127.5.118.130:40959), 9de7cd99b605487e873fb11c786a0a5b (127.5.118.129:43475)
I20260620 14:54:59.171505 24160 raft_consensus.cc:482] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:59.171633 24160 raft_consensus.cc:504] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 } } peers { permanent_uuid: "844b8c55295f4b36b37c853661900d77" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 34893 } } peers { permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 } }
I20260620 14:54:59.172518 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.168740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:40959 (local address 127.5.118.131:44317)
0620 14:54:59.168808 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.168822 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.168862 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.169075 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.169076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.169078 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:59.169153 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.169155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.170131 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.170132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.171405 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.171406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.172174 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.172175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.172288 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.172292 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.172348 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.172418 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:54:59.172525 24160 leader_election.cc:288] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 844b8c55295f4b36b37c853661900d77 (127.5.118.131:34893), 9de7cd99b605487e873fb11c786a0a5b (127.5.118.129:43475)
I20260620 14:54:59.172581 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.168914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:36113 (local address 127.5.118.129:43475)
0620 14:54:59.168951 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.168952 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.169088 (+ 136us) server_negotiation.cc:407] Connection header received
0620 14:54:59.169119 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.169120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.169131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.169182 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:59.169333 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.170112 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.171547 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.172138 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.172239 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.172247 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.172460 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.172511 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:54:59.172799 24149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.168788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:44317 (local address 127.5.118.130:40959)
0620 14:54:59.168838 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.168841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.168871 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:54:59.168990 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.168991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.168999 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.169045 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:59.169249 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.170040 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.171510 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.172059 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.172128 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.172138 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.172703 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.172748 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:54:59.172832 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.168844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:43475 (local address 127.5.118.131:36113)
0620 14:54:59.168985 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.168997 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.169075 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.169202 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.169203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.169205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:59.169269 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.169270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.170190 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.170190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.171397 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.171401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.172287 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.172288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.172366 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.172371 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.172679 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.172736 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":7,"thread_start_us":104,"threads_started":1}
I20260620 14:54:59.173049 23955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bedbd83c745440c0beb79544d3082249" candidate_uuid: "844b8c55295f4b36b37c853661900d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbf69881f4554b86acee25260d8e0151" is_pre_election: true
I20260620 14:54:59.173202 23955 raft_consensus.cc:2454] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 844b8c55295f4b36b37c853661900d77 in term 0.
I20260620 14:54:59.173447 24022 leader_election.cc:302] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 844b8c55295f4b36b37c853661900d77, fbf69881f4554b86acee25260d8e0151; no voters:
I20260620 14:54:59.173425 23821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bedbd83c745440c0beb79544d3082249" candidate_uuid: "844b8c55295f4b36b37c853661900d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de7cd99b605487e873fb11c786a0a5b" is_pre_election: true
I20260620 14:54:59.173579 23821 raft_consensus.cc:2454] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 844b8c55295f4b36b37c853661900d77 in term 0.
I20260620 14:54:59.173583 24158 raft_consensus.cc:2781] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:59.173626 24158 raft_consensus.cc:482] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:59.173653 24158 raft_consensus.cc:3037] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:59.174582 24158 raft_consensus.cc:504] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 } } peers { permanent_uuid: "844b8c55295f4b36b37c853661900d77" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 34893 } } peers { permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 } }
I20260620 14:54:59.174724 24158 leader_election.cc:288] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [CANDIDATE]: Term 1 election: Requested vote from peers fbf69881f4554b86acee25260d8e0151 (127.5.118.130:40959), 9de7cd99b605487e873fb11c786a0a5b (127.5.118.129:43475)
I20260620 14:54:59.174960 23821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bedbd83c745440c0beb79544d3082249" candidate_uuid: "844b8c55295f4b36b37c853661900d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de7cd99b605487e873fb11c786a0a5b"
I20260620 14:54:59.174980 23955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bedbd83c745440c0beb79544d3082249" candidate_uuid: "844b8c55295f4b36b37c853661900d77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbf69881f4554b86acee25260d8e0151"
I20260620 14:54:59.175052 23955 raft_consensus.cc:3037] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:59.175050 23821 raft_consensus.cc:3037] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:59.175971 23955 raft_consensus.cc:2454] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 844b8c55295f4b36b37c853661900d77 in term 1.
I20260620 14:54:59.176041 23821 raft_consensus.cc:2454] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 844b8c55295f4b36b37c853661900d77 in term 1.
I20260620 14:54:59.176230 24022 leader_election.cc:302] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 844b8c55295f4b36b37c853661900d77, fbf69881f4554b86acee25260d8e0151; no voters:
I20260620 14:54:59.176358 24158 raft_consensus.cc:2781] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:59.176442 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.172353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:46629 (local address 127.5.118.131:34893)
0620 14:54:59.172394 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.172396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.172497 (+ 101us) server_negotiation.cc:407] Connection header received
0620 14:54:59.172914 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.172915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.172926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.172960 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:59.173331 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.174032 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.175457 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.176019 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.176074 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.176085 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.176343 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.176386 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:54:59.176476 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.172139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:43475 (local address 127.5.118.130:59129)
0620 14:54:59.172330 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.172341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.172384 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.173002 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.173003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.173004 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:59.173070 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.173072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.174269 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.174270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.175331 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.175334 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.176208 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.176209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.176285 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.176289 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.176337 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.176380 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260620 14:54:59.176517 24158 raft_consensus.cc:686] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 1 LEADER]: Becoming Leader. State: Replica: 844b8c55295f4b36b37c853661900d77, State: Running, Role: LEADER
I20260620 14:54:59.176611 24158 consensus_queue.cc:237] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 } } peers { permanent_uuid: "844b8c55295f4b36b37c853661900d77" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 34893 } } peers { permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 } }
I20260620 14:54:59.176663 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.172151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:34893 (local address 127.5.118.130:46629)
0620 14:54:59.172450 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.172459 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.172516 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.173157 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.173158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.173160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:59.173239 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.173241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.174125 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.174126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.175353 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.175355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.176145 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.176145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.176252 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.176256 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.176543 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.176589 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260620 14:54:59.176713 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.172433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:59129 (local address 127.5.118.129:43475)
0620 14:54:59.172636 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.172638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.172645 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:54:59.172669 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.172669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.172680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.172743 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:59.173157 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.174186 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.175432 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.176060 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.176138 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.176147 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.176572 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.176627 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":66,"threads_started":1}
I20260620 14:54:59.176926 23821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bedbd83c745440c0beb79544d3082249" candidate_uuid: "fbf69881f4554b86acee25260d8e0151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9de7cd99b605487e873fb11c786a0a5b" is_pre_election: true
I20260620 14:54:59.177030 23821 raft_consensus.cc:2379] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fbf69881f4554b86acee25260d8e0151 in current term 1: Already voted for candidate 844b8c55295f4b36b37c853661900d77 in this term.
I20260620 14:54:59.176999 24089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bedbd83c745440c0beb79544d3082249" candidate_uuid: "fbf69881f4554b86acee25260d8e0151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "844b8c55295f4b36b37c853661900d77" is_pre_election: true
I20260620 14:54:59.177323 23889 leader_election.cc:302] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbf69881f4554b86acee25260d8e0151; no voters: 844b8c55295f4b36b37c853661900d77, 9de7cd99b605487e873fb11c786a0a5b
I20260620 14:54:59.177707 24160 raft_consensus.cc:2726] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:54:59.177628 23675 catalog_manager.cc:5495] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 844b8c55295f4b36b37c853661900d77 (127.5.118.131). New cstate: current_term: 1 leader_uuid: "844b8c55295f4b36b37c853661900d77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "844b8c55295f4b36b37c853661900d77" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 34893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 } health_report { overall_health: UNKNOWN } } }
I20260620 14:54:59.212953 24151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.210255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.5.118.131:34893)
0620 14:54:59.210297 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.210300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.210309 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:54:59.210419 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.210420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.210444 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.210540 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:59.210698 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.211184 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.212046 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.212321 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.212412 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.212566 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:59.212676 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:59.212791 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.212834 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.212848 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:54:59.214717 23955 raft_consensus.cc:1261] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 1 FOLLOWER]: Refusing update from remote peer 844b8c55295f4b36b37c853661900d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:59.214715 23821 raft_consensus.cc:1261] T bedbd83c745440c0beb79544d3082249 P 9de7cd99b605487e873fb11c786a0a5b [term 1 FOLLOWER]: Refusing update from remote peer 844b8c55295f4b36b37c853661900d77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:59.215096 24165 consensus_queue.cc:1035] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9de7cd99b605487e873fb11c786a0a5b" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 43475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:59.215240 24158 consensus_queue.cc:1035] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbf69881f4554b86acee25260d8e0151" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:59.220103 24171 mvcc.cc:205] Tried to move back new op lower bound from 7298938057580642304 to 7298938057427881984. Current Snapshot: MvccSnapshot[applied={T|T < 7298938057580642304 or (T in {7298938057580642304})}]
I20260620 14:54:59.220495 23675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:59.220609 23675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:59.221356 23675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18ed9427a89a4f25b7f2ad2334babadb: Sending DeleteTablet for 3 replicas of tablet bedbd83c745440c0beb79544d3082249
I20260620 14:54:59.221709 24069 tablet_service.cc:1514] Processing DeleteTablet for tablet bedbd83c745440c0beb79544d3082249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:59 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:42670
I20260620 14:54:59.221781 23935 tablet_service.cc:1514] Processing DeleteTablet for tablet bedbd83c745440c0beb79544d3082249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:59 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:58520
I20260620 14:54:59.221793 23801 tablet_service.cc:1514] Processing DeleteTablet for tablet bedbd83c745440c0beb79544d3082249 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:59 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:37550
I20260620 14:54:59.221967 24177 tablet_replica.cc:330] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: stopping tablet replica
I20260620 14:54:59.221992 24178 tablet_replica.cc:330] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: stopping tablet replica
I20260620 14:54:59.222074 24177 raft_consensus.cc:2229] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:54:59.222132 24178 raft_consensus.cc:2229] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:59.222184 24177 raft_consensus.cc:2258] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:59.222285 24178 raft_consensus.cc:2258] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:59.222369 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 23738
I20260620 14:54:59.222656 24177 ts_tablet_manager.cc:1905] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:59.222764 24178 ts_tablet_manager.cc:1905] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:59.224606 24178 ts_tablet_manager.cc:1918] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:59.224706 24178 log.cc:1196] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/bedbd83c745440c0beb79544d3082249
I20260620 14:54:59.224694 24177 ts_tablet_manager.cc:1918] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:59.224761 24177 log.cc:1196] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/bedbd83c745440c0beb79544d3082249
I20260620 14:54:59.225040 24177 ts_tablet_manager.cc:1939] T bedbd83c745440c0beb79544d3082249 P 844b8c55295f4b36b37c853661900d77: Deleting consensus metadata
I20260620 14:54:59.225035 24178 ts_tablet_manager.cc:1939] T bedbd83c745440c0beb79544d3082249 P fbf69881f4554b86acee25260d8e0151: Deleting consensus metadata
I20260620 14:54:59.225454 23661 catalog_manager.cc:4841] TS 844b8c55295f4b36b37c853661900d77 (127.5.118.131:34893): tablet bedbd83c745440c0beb79544d3082249 (table test-table [id=8881ff0f3f5b4d4f9b4e8249ce6c595a]) successfully deleted
I20260620 14:54:59.225492 23660 catalog_manager.cc:4841] TS fbf69881f4554b86acee25260d8e0151 (127.5.118.130:40959): tablet bedbd83c745440c0beb79544d3082249 (table test-table [id=8881ff0f3f5b4d4f9b4e8249ce6c595a]) successfully deleted
W20260620 14:54:59.233646 23663 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:43475 (error 108)
I20260620 14:54:59.233805 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 23872
I20260620 14:54:59.234658 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.234513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:43475 (local address 127.0.0.1:37560)
0620 14:54:59.234573 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.234612 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:43475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:54:59.234823 23663 catalog_manager.cc:4584] TS 9de7cd99b605487e873fb11c786a0a5b (127.5.118.129:43475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bedbd83c745440c0beb79544d3082249: Network error: Client connection negotiation failed: client connection to 127.5.118.129:43475: connect: Connection refused (error 111)
I20260620 14:54:59.240077 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 24006
I20260620 14:54:59.244299 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 23644
2026-06-20T14:54:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2435 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T14:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:54:59Z Disabled control of system clock
I20260620 14:54:59.272524 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:40591
--webserver_interface=127.5.118.190
--webserver_port=0
--builtin_ntp_servers=127.5.118.148:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:40591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:54:59.355024 24184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:59.355257 24184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:59.355278 24184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:59.356498 24184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:54:59.356557 24184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:59.356578 24184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:59.356599 24184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:54:59.356611 24184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:54:59.358007 24184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:40591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:40591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:59.358224 24184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:59.358484 24184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:59.363598 24192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:59.363750 24184 server_base.cc:1034] running on GCE node
W20260620 14:54:59.363579 24190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:59.363583 24189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:59.364276 24184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:59.364625 24184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:59.365803 24184 hybrid_clock.cc:648] HybridClock initialized: now 1781967299365801 us; error 39 us; skew 500 ppm
I20260620 14:54:59.367158 24184 webserver.cc:460] Webserver started at http://127.5.118.190:38003/ using document root <none> and password file <none>
I20260620 14:54:59.367451 24184 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:59.367518 24184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:59.367648 24184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:59.368813 24184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "72aa216097cc438b8a31d17b944d170d"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "08149a9bec72650571260db4346e7d3d"
server_key_iv: "a8cace600e0408dab05af98028f7dcf4"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.369194 24184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "72aa216097cc438b8a31d17b944d170d"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "08149a9bec72650571260db4346e7d3d"
server_key_iv: "a8cace600e0408dab05af98028f7dcf4"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.370662 24184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:54:59.371465 24198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.371662 24184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:59.371732 24184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "72aa216097cc438b8a31d17b944d170d"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "08149a9bec72650571260db4346e7d3d"
server_key_iv: "a8cace600e0408dab05af98028f7dcf4"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.371840 24184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:59.386397 24184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:59.386788 24184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:59.386963 24184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:59.391008 24184 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:40591
I20260620 14:54:59.391044 24250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:40591 every 8 connection(s)
I20260620 14:54:59.391424 24184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:54:59.392581 24251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:59.394909 24251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d: Bootstrap starting.
I20260620 14:54:59.395442 24251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:59.395681 24251 log.cc:824] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:59.396463 24251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d: No bootstrap required, opened a new log
I20260620 14:54:59.397238 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 24184
I20260620 14:54:59.397342 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:54:59.397548 5594 external_mini_cluster.cc:1428] Setting key 223eb0b1c6584f2f5b0c279e1e445717
I20260620 14:54:59.398715 24251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72aa216097cc438b8a31d17b944d170d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40591 } }
I20260620 14:54:59.398883 24251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:59.398931 24251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72aa216097cc438b8a31d17b944d170d, State: Initialized, Role: FOLLOWER
I20260620 14:54:59.399096 24251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72aa216097cc438b8a31d17b944d170d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40591 } }
I20260620 14:54:59.399192 24251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:54:59.399219 24251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:54:59.399289 24251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:59.399832 24251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72aa216097cc438b8a31d17b944d170d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40591 } }
I20260620 14:54:59.399950 24251 leader_election.cc:302] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72aa216097cc438b8a31d17b944d170d; no voters:
I20260620 14:54:59.400141 24251 leader_election.cc:288] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:54:59.400198 24256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:59.400300 24256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [term 1 LEADER]: Becoming Leader. State: Replica: 72aa216097cc438b8a31d17b944d170d, State: Running, Role: LEADER
I20260620 14:54:59.400383 24256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72aa216097cc438b8a31d17b944d170d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40591 } }
I20260620 14:54:59.400445 24251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:54:59.400749 24256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72aa216097cc438b8a31d17b944d170d. Latest consensus state: current_term: 1 leader_uuid: "72aa216097cc438b8a31d17b944d170d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72aa216097cc438b8a31d17b944d170d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40591 } } }
I20260620 14:54:59.400828 24256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:59.400758 24257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72aa216097cc438b8a31d17b944d170d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72aa216097cc438b8a31d17b944d170d" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 40591 } } }
I20260620 14:54:59.400875 24257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d [sys.catalog]: This master's current role is: LEADER
I20260620 14:54:59.401156 24261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:54:59.401691 24261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:54:59.403690 24261 catalog_manager.cc:1347] Generated new cluster ID: 320320c717154500baa9e25f5fc9057e
I20260620 14:54:59.403754 24261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:54:59.404412 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.398099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42382 (local address 127.5.118.190:40591)
0620 14:54:59.398322 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.398329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.398354 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:54:59.398692 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.398698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.398758 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.398873 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:59.399134 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.400821 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.401978 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.402573 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.403076 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.403480 (+ 404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:59.403484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:59.403484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:59.403717 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:59.403781 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:59.403799 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:59.403890 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.403970 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.404146 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260620 14:54:59.416040 24261 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:54:59.416880 24261 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:54:59.425518 24261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d: Generated new TSK 0
I20260620 14:54:59.425743 24261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:54:59.429277 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40591
--builtin_ntp_servers=127.5.118.148:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:54:59.512485 24275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:59.512732 24275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:59.512768 24275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:59.513999 24275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:59.514072 24275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:59.514119 24275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:54:59.515573 24275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:59.515836 24275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:59.516211 24275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:59.521345 24280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:59.521330 24281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:59.521663 24275 server_base.cc:1034] running on GCE node
W20260620 14:54:59.521346 24283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:59.521927 24275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:59.522181 24275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:59.523363 24275 hybrid_clock.cc:648] HybridClock initialized: now 1781967299523350 us; error 30 us; skew 500 ppm
I20260620 14:54:59.524715 24275 webserver.cc:460] Webserver started at http://127.5.118.129:44365/ using document root <none> and password file <none>
I20260620 14:54:59.524986 24275 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:59.525064 24275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:59.525213 24275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:59.526336 24275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "0dab6d9ae3e0425f95120c278c9d90b5"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "450bb4597ae9c044bf51ab97c6463fd6"
server_key_iv: "e01756a2ff8fe981299409e29b0d3ce1"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.526718 24275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "0dab6d9ae3e0425f95120c278c9d90b5"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "450bb4597ae9c044bf51ab97c6463fd6"
server_key_iv: "e01756a2ff8fe981299409e29b0d3ce1"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.528254 24275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 14:54:59.529201 24289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.529407 24275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:54:59.529548 24275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "0dab6d9ae3e0425f95120c278c9d90b5"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "450bb4597ae9c044bf51ab97c6463fd6"
server_key_iv: "e01756a2ff8fe981299409e29b0d3ce1"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.529639 24275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:59.542202 24275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:59.542557 24275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:59.542716 24275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:59.542953 24275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:59.543285 24275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:59.543339 24275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.543381 24275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:59.543407 24275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.549445 24275 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:39891
I20260620 14:54:59.549507 24402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:39891 every 8 connection(s)
I20260620 14:54:59.549872 24275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:54:59.554807 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 24275
I20260620 14:54:59.554937 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:54:59.555171 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.550642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:46787 (local address 127.5.118.190:40591)
0620 14:54:59.550735 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.550738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.550982 (+ 244us) server_negotiation.cc:407] Connection header received
0620 14:54:59.551486 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.551488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.551500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.551551 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:59.552234 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.552748 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.553754 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.553936 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.553996 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.554770 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:59.554772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:59.554773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:59.554779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:59.554797 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:59.554802 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:59.554863 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.555064 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.555094 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:54:59.555188 5594 external_mini_cluster.cc:1428] Setting key 6f219e7350c3ea6e957b81bdec6c15fc
I20260620 14:54:59.555361 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.550605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40591 (local address 127.5.118.129:46787)
0620 14:54:59.550817 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.550852 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.551376 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.551613 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.551617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.551626 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:59.552152 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.552157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.552837 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.552838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.553676 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.553679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.554015 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.554016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.554127 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.554622 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:59.554622 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:59.554651 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:59.554674 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:59.554868 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:59.554872 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:59.554897 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.555020 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.555130 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":102,"threads_started":1}
I20260620 14:54:59.555913 24403 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40591
I20260620 14:54:59.556016 24403 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:59.556245 24403 heartbeater.cc:511] Master 127.5.118.190:40591 requested a full tablet report, sending...
I20260620 14:54:59.556255 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40591
--builtin_ntp_servers=127.5.118.148:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:54:59.556744 24215 ts_manager.cc:194] Registered new tserver with Master: 0dab6d9ae3e0425f95120c278c9d90b5 (127.5.118.129:39891)
I20260620 14:54:59.557466 24215 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.129:46787
W20260620 14:54:59.638165 24406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:59.638392 24406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:59.638414 24406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:59.639683 24406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:59.639734 24406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:59.639760 24406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:54:59.641291 24406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:59.641556 24406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:59.641862 24406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:54:59.647620 24412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:59.647676 24414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:59.647624 24411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:59.648895 24406 server_base.cc:1034] running on GCE node
I20260620 14:54:59.649153 24406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:59.649420 24406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:59.650591 24406 hybrid_clock.cc:648] HybridClock initialized: now 1781967299650589 us; error 40 us; skew 500 ppm
I20260620 14:54:59.651835 24406 webserver.cc:460] Webserver started at http://127.5.118.130:36389/ using document root <none> and password file <none>
I20260620 14:54:59.652096 24406 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:59.652166 24406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:59.652349 24406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:59.653504 24406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "c2f0edd852a7421fbac70566818072c9"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "a2c67c592f37b7e033631791cc59e8ae"
server_key_iv: "98db67dd3ee34aded34561b1b1a96a99"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.653868 24406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "c2f0edd852a7421fbac70566818072c9"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "a2c67c592f37b7e033631791cc59e8ae"
server_key_iv: "98db67dd3ee34aded34561b1b1a96a99"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.655184 24406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:54:59.655954 24420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.656145 24406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 14:54:59.656275 24406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "c2f0edd852a7421fbac70566818072c9"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "a2c67c592f37b7e033631791cc59e8ae"
server_key_iv: "98db67dd3ee34aded34561b1b1a96a99"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.656369 24406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:59.677062 24406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:59.677423 24406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:59.677587 24406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:59.677840 24406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:59.678196 24406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:59.678253 24406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.678297 24406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:59.678328 24406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.684512 24406 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:35875
I20260620 14:54:59.684552 24533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:35875 every 8 connection(s)
I20260620 14:54:59.684916 24406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:54:59.690356 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.685681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:57971 (local address 127.5.118.190:40591)
0620 14:54:59.685752 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.685754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.685997 (+ 243us) server_negotiation.cc:407] Connection header received
0620 14:54:59.686554 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.686555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.686567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.686622 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:59.687308 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.687816 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.688855 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.689079 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.689167 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.690037 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:59.690038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:59.690039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:59.690044 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:59.690061 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:59.690066 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:59.690141 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.690236 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.690270 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:54:59.690588 24536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.685663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40591 (local address 127.5.118.130:57971)
0620 14:54:59.685867 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.685903 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.686442 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.686707 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.686711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.686719 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:59.687213 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.687217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.687920 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.687921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.688746 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.688748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.689180 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.689181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.689320 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.689888 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:59.689888 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:59.689920 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:59.689945 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:59.690141 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:59.690148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:59.690151 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.690240 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.690352 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260620 14:54:59.691143 24534 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40591
I20260620 14:54:59.691267 24534 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:59.691483 24534 heartbeater.cc:511] Master 127.5.118.190:40591 requested a full tablet report, sending...
I20260620 14:54:59.691514 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 24406
I20260620 14:54:59.691628 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:54:59.691820 5594 external_mini_cluster.cc:1428] Setting key 88ec5673051d9dca19493dbbe673c284
I20260620 14:54:59.691932 24215 ts_manager.cc:194] Registered new tserver with Master: c2f0edd852a7421fbac70566818072c9 (127.5.118.130:35875)
I20260620 14:54:59.692425 24215 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.130:57971
I20260620 14:54:59.692930 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40591
--builtin_ntp_servers=127.5.118.148:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:54:59.779855 24537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:54:59.780102 24537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:54:59.780141 24537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:54:59.781358 24537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:54:59.781426 24537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:54:59.781469 24537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:54:59.782960 24537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:40591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:54:59.783257 24537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:54:59.783589 24537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:54:59.789184 24537 server_base.cc:1034] running on GCE node
W20260620 14:54:59.789129 24542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:59.789222 24543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:54:59.789135 24545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:54:59.789506 24537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:54:59.789810 24537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:54:59.791015 24537 hybrid_clock.cc:648] HybridClock initialized: now 1781967299790997 us; error 29 us; skew 500 ppm
I20260620 14:54:59.792486 24537 webserver.cc:460] Webserver started at http://127.5.118.131:44843/ using document root <none> and password file <none>
I20260620 14:54:59.792764 24537 fs_manager.cc:359] Metadata directory not provided
I20260620 14:54:59.792841 24537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:54:59.793009 24537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:54:59.794117 24537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "c30a9def809c4537811d0b029d7dbb09"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "d97b4616963db41c4434461807481f0e"
server_key_iv: "e599742bcadf1ed88b73d6241d6f1f68"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.794486 24537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "c30a9def809c4537811d0b029d7dbb09"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "d97b4616963db41c4434461807481f0e"
server_key_iv: "e599742bcadf1ed88b73d6241d6f1f68"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.795837 24537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:54:59.796710 24551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.796900 24537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:54:59.797010 24537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "c30a9def809c4537811d0b029d7dbb09"
format_stamp: "Formatted at 2026-06-20 14:54:59 on dist-test-slave-2m5f"
server_key: "d97b4616963db41c4434461807481f0e"
server_key_iv: "e599742bcadf1ed88b73d6241d6f1f68"
server_key_version: "encryptionkey@0"
I20260620 14:54:59.797101 24537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:54:59.805711 24537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:54:59.806033 24537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:54:59.806185 24537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:54:59.806411 24537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:54:59.806736 24537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:54:59.806792 24537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.806834 24537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:54:59.806862 24537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:54:59.813024 24537 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:36457
I20260620 14:54:59.813062 24664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:36457 every 8 connection(s)
I20260620 14:54:59.813433 24537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:54:59.817832 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 24537
I20260620 14:54:59.817991 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:54:59.818199 5594 external_mini_cluster.cc:1428] Setting key f3516c3cbc179e366e1e6c322d623524
I20260620 14:54:59.819372 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.814307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:39021 (local address 127.5.118.190:40591)
0620 14:54:59.814431 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.814433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.814630 (+ 197us) server_negotiation.cc:407] Connection header received
0620 14:54:59.815183 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.815184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.815198 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.815251 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:59.815998 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.816540 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.817579 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.817782 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.817859 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.819039 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:59.819040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:59.819041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:59.819046 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:59.819062 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:59.819066 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:59.819139 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.819238 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.819290 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:54:59.819598 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.814208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:40591 (local address 127.5.118.131:39021)
0620 14:54:59.814493 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.814541 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.815071 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.815340 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.815347 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.815361 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:54:59.815873 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.815878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.816648 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.816649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.817469 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.817472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.817867 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.817868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.818065 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.818779 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:54:59.818780 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:54:59.818813 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:54:59.818841 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:54:59.819140 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:54:59.819146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:54:59.819150 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.819234 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.819346 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
I20260620 14:54:59.820080 24665 heartbeater.cc:347] Connected to a master server at 127.5.118.190:40591
I20260620 14:54:59.820209 24665 heartbeater.cc:464] Registering TS with master...
I20260620 14:54:59.820403 24665 heartbeater.cc:511] Master 127.5.118.190:40591 requested a full tablet report, sending...
I20260620 14:54:59.820801 24215 ts_manager.cc:194] Registered new tserver with Master: c30a9def809c4537811d0b029d7dbb09 (127.5.118.131:36457)
I20260620 14:54:59.821164 24215 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.131:39021
I20260620 14:54:59.830076 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:54:59.834576 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.831538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.5.118.190:40591)
0620 14:54:59.831602 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.831604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.831731 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:54:59.831811 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.831811 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.831823 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.831876 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:54:59.832042 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.832601 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.833403 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.833591 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.833647 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.834368 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:54:59.834369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:54:59.834369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:54:59.834373 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:54:59.834386 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:54:59.834390 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:54:59.834462 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.834508 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.834518 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:54:59.836344 24215 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:54:59.837893 24215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:54:59.844579 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.840454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:36457 (local address 127.0.0.1:58906)
0620 14:54:59.840759 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.840767 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.840806 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.841400 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.841400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.841402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:59.841452 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.841454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.842289 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.842291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.843324 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.843327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.844226 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.844227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.844308 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.844325 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.844401 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.844456 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260620 14:54:59.844704 24673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.840220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:35875 (local address 127.0.0.1:46202)
0620 14:54:59.840543 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.840591 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.840660 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.841119 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.841120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.841122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:59.841156 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.841157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.842394 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.842395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.843642 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.843644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.844406 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.844407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.844472 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.844477 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.844543 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.844596 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":44,"thread_start_us":74,"threads_started":1}
I20260620 14:54:59.844864 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.840557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58906 (local address 127.5.118.131:36457)
0620 14:54:59.841238 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.841242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.841251 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:54:59.841287 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.841289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.841301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.841400 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:59.841515 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.842194 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.843420 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.844095 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.844196 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.844293 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.844712 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.844767 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":64,"threads_started":1}
I20260620 14:54:59.844903 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.840220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39891 (local address 127.0.0.1:51266)
0620 14:54:59.840592 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.840600 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.840660 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.840869 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.840872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.840874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:59.840949 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.840950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.842366 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.842367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.843613 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.843615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.844688 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.844689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.844750 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.844753 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.844813 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.844849 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":37,"thread_start_us":95,"threads_started":1}
I20260620 14:54:59.844964 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.840477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.5.118.129:39891)
0620 14:54:59.840676 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.840681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.840695 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:54:59.840746 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.840748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.840760 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.840834 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:59.841040 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.842248 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.843707 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.844453 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.844519 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.844652 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.844830 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.844867 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":89,"threads_started":1}
I20260620 14:54:59.845759 24599 tablet_service.cc:1467] Processing CreateTablet for tablet d3f25028c5ff4f4fa2f4e5badcca37fc (DEFAULT_TABLE table=test-table [id=23df8f6fd8c44e959bf33dc9e2a4c5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:59.845855 24337 tablet_service.cc:1467] Processing CreateTablet for tablet d3f25028c5ff4f4fa2f4e5badcca37fc (DEFAULT_TABLE table=test-table [id=23df8f6fd8c44e959bf33dc9e2a4c5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:59.846172 24599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f25028c5ff4f4fa2f4e5badcca37fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:59.846187 24337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f25028c5ff4f4fa2f4e5badcca37fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:59.848816 24679 tablet_bootstrap.cc:492] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5: Bootstrap starting.
I20260620 14:54:59.848924 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.840348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.5.118.130:35875)
0620 14:54:59.840962 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.840967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.840977 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:54:59.841016 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.841019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.841034 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.841111 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:59.841238 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.842315 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.843728 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.844319 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.844389 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.844487 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.844557 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.844596 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":90,"threads_started":1}
I20260620 14:54:59.849320 24680 tablet_bootstrap.cc:492] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09: Bootstrap starting.
I20260620 14:54:59.849531 24679 tablet_bootstrap.cc:654] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:59.849606 24468 tablet_service.cc:1467] Processing CreateTablet for tablet d3f25028c5ff4f4fa2f4e5badcca37fc (DEFAULT_TABLE table=test-table [id=23df8f6fd8c44e959bf33dc9e2a4c5da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:54:59.849859 24679 log.cc:824] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:59.849974 24468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f25028c5ff4f4fa2f4e5badcca37fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:54:59.850029 24680 tablet_bootstrap.cc:654] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:59.850356 24680 log.cc:824] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:59.850697 24679 tablet_bootstrap.cc:492] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5: No bootstrap required, opened a new log
I20260620 14:54:59.850769 24679 ts_tablet_manager.cc:1397] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:54:59.851076 24680 tablet_bootstrap.cc:492] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09: No bootstrap required, opened a new log
I20260620 14:54:59.851158 24680 ts_tablet_manager.cc:1397] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:54:59.853257 24680 raft_consensus.cc:348] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f0edd852a7421fbac70566818072c9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35875 } } peers { permanent_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39891 } } peers { permanent_uuid: "c30a9def809c4537811d0b029d7dbb09" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36457 } }
I20260620 14:54:59.853466 24680 raft_consensus.cc:374] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:59.853515 24680 raft_consensus.cc:729] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30a9def809c4537811d0b029d7dbb09, State: Initialized, Role: FOLLOWER
I20260620 14:54:59.853549 24679 raft_consensus.cc:348] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f0edd852a7421fbac70566818072c9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35875 } } peers { permanent_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39891 } } peers { permanent_uuid: "c30a9def809c4537811d0b029d7dbb09" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36457 } }
I20260620 14:54:59.853649 24683 tablet_bootstrap.cc:492] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: Bootstrap starting.
I20260620 14:54:59.853626 24680 consensus_queue.cc:260] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f0edd852a7421fbac70566818072c9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35875 } } peers { permanent_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39891 } } peers { permanent_uuid: "c30a9def809c4537811d0b029d7dbb09" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36457 } }
I20260620 14:54:59.853732 24679 raft_consensus.cc:374] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:59.853778 24679 raft_consensus.cc:729] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dab6d9ae3e0425f95120c278c9d90b5, State: Initialized, Role: FOLLOWER
I20260620 14:54:59.853933 24680 ts_tablet_manager.cc:1428] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:54:59.853919 24679 consensus_queue.cc:260] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f0edd852a7421fbac70566818072c9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35875 } } peers { permanent_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39891 } } peers { permanent_uuid: "c30a9def809c4537811d0b029d7dbb09" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36457 } }
I20260620 14:54:59.854086 24665 heartbeater.cc:503] Master 127.5.118.190:40591 was elected leader, sending a full tablet report...
I20260620 14:54:59.854382 24683 tablet_bootstrap.cc:654] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: Neither blocks nor log segments found. Creating new log.
I20260620 14:54:59.854704 24683 log.cc:824] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:54:59.854840 24403 heartbeater.cc:503] Master 127.5.118.190:40591 was elected leader, sending a full tablet report...
I20260620 14:54:59.855103 24679 ts_tablet_manager.cc:1428] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 14:54:59.855506 24683 tablet_bootstrap.cc:492] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: No bootstrap required, opened a new log
I20260620 14:54:59.855592 24683 ts_tablet_manager.cc:1397] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:54:59.857579 24683 raft_consensus.cc:348] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f0edd852a7421fbac70566818072c9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35875 } } peers { permanent_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39891 } } peers { permanent_uuid: "c30a9def809c4537811d0b029d7dbb09" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36457 } }
I20260620 14:54:59.857743 24683 raft_consensus.cc:374] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:54:59.857793 24683 raft_consensus.cc:729] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f0edd852a7421fbac70566818072c9, State: Initialized, Role: FOLLOWER
I20260620 14:54:59.857929 24683 consensus_queue.cc:260] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f0edd852a7421fbac70566818072c9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35875 } } peers { permanent_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39891 } } peers { permanent_uuid: "c30a9def809c4537811d0b029d7dbb09" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36457 } }
I20260620 14:54:59.858194 24683 ts_tablet_manager.cc:1428] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 14:54:59.858283 24534 heartbeater.cc:503] Master 127.5.118.190:40591 was elected leader, sending a full tablet report...
I20260620 14:54:59.910246 24685 raft_consensus.cc:482] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:54:59.910373 24685 raft_consensus.cc:504] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f0edd852a7421fbac70566818072c9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35875 } } peers { permanent_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39891 } } peers { permanent_uuid: "c30a9def809c4537811d0b029d7dbb09" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36457 } }
I20260620 14:54:59.910782 24685 leader_election.cc:288] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2f0edd852a7421fbac70566818072c9 (127.5.118.130:35875), c30a9def809c4537811d0b029d7dbb09 (127.5.118.131:36457)
I20260620 14:54:59.914067 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.911092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:35875 (local address 127.5.118.129:42361)
0620 14:54:59.911157 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.911170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.911257 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.911436 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.911437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.911439 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:59.911533 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.911536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.912366 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.912367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.913182 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.913184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.913786 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.913787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.913862 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.913865 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.913946 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.913987 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":17}
I20260620 14:54:59.914064 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.911203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:42361 (local address 127.5.118.130:35875)
0620 14:54:59.911257 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.911259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.911273 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:54:59.911323 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.911324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.911337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.911407 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:59.911624 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.912242 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.913275 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.913718 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.913785 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.913794 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.913956 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.913994 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:54:59.914139 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.911304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:54343 (local address 127.5.118.131:36457)
0620 14:54:59.911352 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.911353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.911474 (+ 121us) server_negotiation.cc:407] Connection header received
0620 14:54:59.911526 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.911527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.911537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.911598 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:54:59.911755 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.912358 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.913335 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.913681 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.913769 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.913778 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.914009 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.914057 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 14:54:59.914390 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.911205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:36457 (local address 127.5.118.129:54343)
0620 14:54:59.911401 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.911412 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:54:59.911476 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:54:59.911622 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:54:59.911623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:54:59.911625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:54:59.911690 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.911691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.912442 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.912442 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.913209 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:54:59.913211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.913828 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.913829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:54:59.913921 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.913925 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:54:59.914264 (+ 339us) client_negotiation.cc:240] Negotiation successful
0620 14:54:59.914315 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":129,"threads_started":1}
I20260620 14:54:59.914440 24488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f25028c5ff4f4fa2f4e5badcca37fc" candidate_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2f0edd852a7421fbac70566818072c9" is_pre_election: true
I20260620 14:54:59.914652 24488 raft_consensus.cc:2454] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dab6d9ae3e0425f95120c278c9d90b5 in term 0.
I20260620 14:54:59.914754 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f25028c5ff4f4fa2f4e5badcca37fc" candidate_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30a9def809c4537811d0b029d7dbb09" is_pre_election: true
I20260620 14:54:59.914930 24619 raft_consensus.cc:2454] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dab6d9ae3e0425f95120c278c9d90b5 in term 0.
I20260620 14:54:59.914902 24292 leader_election.cc:302] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dab6d9ae3e0425f95120c278c9d90b5, c2f0edd852a7421fbac70566818072c9; no voters:
I20260620 14:54:59.915053 24685 raft_consensus.cc:2781] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:54:59.915107 24685 raft_consensus.cc:482] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:54:59.915131 24685 raft_consensus.cc:3037] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:59.915832 24685 raft_consensus.cc:504] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f0edd852a7421fbac70566818072c9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35875 } } peers { permanent_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39891 } } peers { permanent_uuid: "c30a9def809c4537811d0b029d7dbb09" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36457 } }
I20260620 14:54:59.915984 24685 leader_election.cc:288] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [CANDIDATE]: Term 1 election: Requested vote from peers c2f0edd852a7421fbac70566818072c9 (127.5.118.130:35875), c30a9def809c4537811d0b029d7dbb09 (127.5.118.131:36457)
I20260620 14:54:59.916189 24619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f25028c5ff4f4fa2f4e5badcca37fc" candidate_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30a9def809c4537811d0b029d7dbb09"
I20260620 14:54:59.916215 24488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f25028c5ff4f4fa2f4e5badcca37fc" candidate_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2f0edd852a7421fbac70566818072c9"
I20260620 14:54:59.916275 24619 raft_consensus.cc:3037] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:59.916298 24488 raft_consensus.cc:3037] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:54:59.917006 24619 raft_consensus.cc:2454] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dab6d9ae3e0425f95120c278c9d90b5 in term 1.
I20260620 14:54:59.917132 24488 raft_consensus.cc:2454] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dab6d9ae3e0425f95120c278c9d90b5 in term 1.
I20260620 14:54:59.917176 24292 leader_election.cc:302] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dab6d9ae3e0425f95120c278c9d90b5, c30a9def809c4537811d0b029d7dbb09; no voters:
I20260620 14:54:59.917256 24685 raft_consensus.cc:2781] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:54:59.917410 24685 raft_consensus.cc:686] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [term 1 LEADER]: Becoming Leader. State: Replica: 0dab6d9ae3e0425f95120c278c9d90b5, State: Running, Role: LEADER
I20260620 14:54:59.917524 24685 consensus_queue.cc:237] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f0edd852a7421fbac70566818072c9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35875 } } peers { permanent_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39891 } } peers { permanent_uuid: "c30a9def809c4537811d0b029d7dbb09" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36457 } }
I20260620 14:54:59.918457 24215 catalog_manager.cc:5495] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dab6d9ae3e0425f95120c278c9d90b5 (127.5.118.129). New cstate: current_term: 1 leader_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2f0edd852a7421fbac70566818072c9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dab6d9ae3e0425f95120c278c9d90b5" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 39891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c30a9def809c4537811d0b029d7dbb09" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36457 } health_report { overall_health: UNKNOWN } } }
W20260620 14:54:59.935514 24535 tablet.cc:2369] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:54:59.938123 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.935423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.5.118.129:39891)
0620 14:54:59.935495 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 14:54:59.935497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:54:59.935504 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:54:59.935540 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:54:59.935541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:54:59.935561 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:54:59.935647 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:54:59.935911 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.936486 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.937304 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:54:59.937499 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:54:59.937579 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:54:59.937747 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:54:59.937853 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:54:59.937972 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:54:59.938010 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 14:54:59.938022 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:54:59.940444 24619 raft_consensus.cc:1261] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09 [term 1 FOLLOWER]: Refusing update from remote peer 0dab6d9ae3e0425f95120c278c9d90b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:59.940491 24488 raft_consensus.cc:1261] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9 [term 1 FOLLOWER]: Refusing update from remote peer 0dab6d9ae3e0425f95120c278c9d90b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:54:59.940924 24689 consensus_queue.cc:1035] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30a9def809c4537811d0b029d7dbb09" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:59.941089 24685 consensus_queue.cc:1035] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2f0edd852a7421fbac70566818072c9" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 35875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:54:59.944576 24694 mvcc.cc:205] Tried to move back new op lower bound from 7298938060552368128 to 7298938060462735360. Current Snapshot: MvccSnapshot[applied={T|T < 7298938060552368128 or (T in {7298938060552368128})}]
I20260620 14:54:59.944792 24693 mvcc.cc:205] Tried to move back new op lower bound from 7298938060552368128 to 7298938060462735360. Current Snapshot: MvccSnapshot[applied={T|T < 7298938060552368128}]
I20260620 14:54:59.946169 24214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:59.946300 24214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:54:59.947201 24214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72aa216097cc438b8a31d17b944d170d: Sending DeleteTablet for 3 replicas of tablet d3f25028c5ff4f4fa2f4e5badcca37fc
I20260620 14:54:59.947587 24468 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f25028c5ff4f4fa2f4e5badcca37fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:59 UTC) from {username='slave'} at 127.0.0.1:46202
I20260620 14:54:59.947856 24702 tablet_replica.cc:330] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: stopping tablet replica
I20260620 14:54:59.947973 24702 raft_consensus.cc:2229] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:59.947929 24337 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f25028c5ff4f4fa2f4e5badcca37fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:59 UTC) from {username='slave'} at 127.0.0.1:51266
I20260620 14:54:59.948086 24702 raft_consensus.cc:2258] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:59.948117 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 24275
I20260620 14:54:59.948154 24703 tablet_replica.cc:330] T d3f25028c5ff4f4fa2f4e5badcca37fc P 0dab6d9ae3e0425f95120c278c9d90b5: stopping tablet replica
I20260620 14:54:59.949173 24702 ts_tablet_manager.cc:1905] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:59.949833 24599 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f25028c5ff4f4fa2f4e5badcca37fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:54:59 UTC) from {username='slave'} at 127.0.0.1:58906
I20260620 14:54:59.950176 24704 tablet_replica.cc:330] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09: stopping tablet replica
I20260620 14:54:59.950301 24704 raft_consensus.cc:2229] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:54:59.950417 24704 raft_consensus.cc:2258] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:54:59.951078 24702 ts_tablet_manager.cc:1918] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:59.951165 24702 log.cc:1196] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/d3f25028c5ff4f4fa2f4e5badcca37fc
I20260620 14:54:59.951417 24702 ts_tablet_manager.cc:1939] T d3f25028c5ff4f4fa2f4e5badcca37fc P c2f0edd852a7421fbac70566818072c9: Deleting consensus metadata
I20260620 14:54:59.951900 24201 catalog_manager.cc:4841] TS c2f0edd852a7421fbac70566818072c9 (127.5.118.130:35875): tablet d3f25028c5ff4f4fa2f4e5badcca37fc (table test-table [id=23df8f6fd8c44e959bf33dc9e2a4c5da]) successfully deleted
I20260620 14:54:59.952601 24704 ts_tablet_manager.cc:1905] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:54:59.954571 24704 ts_tablet_manager.cc:1918] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:54:59.954663 24704 log.cc:1196] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/d3f25028c5ff4f4fa2f4e5badcca37fc
I20260620 14:54:59.954949 24704 ts_tablet_manager.cc:1939] T d3f25028c5ff4f4fa2f4e5badcca37fc P c30a9def809c4537811d0b029d7dbb09: Deleting consensus metadata
W20260620 14:54:59.955030 24200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:39891 (error 108)
I20260620 14:54:59.955399 24201 catalog_manager.cc:4841] TS c30a9def809c4537811d0b029d7dbb09 (127.5.118.131:36457): tablet d3f25028c5ff4f4fa2f4e5badcca37fc (table test-table [id=23df8f6fd8c44e959bf33dc9e2a4c5da]) successfully deleted
I20260620 14:54:59.955611 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 24406
I20260620 14:54:59.955631 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:54:59.955503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:39891 (local address 127.0.0.1:51278)
0620 14:54:59.955550 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:54:59.955599 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 14:54:59.955776 24200 catalog_manager.cc:4584] TS 0dab6d9ae3e0425f95120c278c9d90b5 (127.5.118.129:39891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3f25028c5ff4f4fa2f4e5badcca37fc: Network error: Client connection negotiation failed: client connection to 127.5.118.129:39891: connect: Connection refused (error 111)
I20260620 14:54:59.961373 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 24537
I20260620 14:54:59.966200 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 24184
2026-06-20T14:54:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (722 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:00 dist-test-slave-2m5f krb5kdc[24708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:00 dist-test-slave-2m5f krb5kdc[24708](info): set up 2 sockets
Jun 20 14:55:00 dist-test-slave-2m5f krb5kdc[24708](info): commencing operation
krb5kdc: starting...
W20260620 14:55:01.044638 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:01 dist-test-slave-2m5f krb5kdc[24708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:55:01.171973 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:32915
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:01.255788 24721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:01.256040 24721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:01.256067 24721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:01.257421 24721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:01.257484 24721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:01.257499 24721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:01.257513 24721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:01.257525 24721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:01.259095 24721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:01.259326 24721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:01.259527 24721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:01.264887 24727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:01.264887 24726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:01.265029 24729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:01.265298 24721 server_base.cc:1034] running on GCE node
I20260620 14:55:01.265561 24721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:01.265827 24721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:01.267038 24721 hybrid_clock.cc:648] HybridClock initialized: now 1781967301267025 us; error 40 us; skew 500 ppm
Jun 20 14:55:01 dist-test-slave-2m5f krb5kdc[24708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967301, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:01.269836 24721 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:55:01.270354 24721 webserver.cc:460] Webserver started at http://127.5.118.190:45033/ using document root <none> and password file <none>
I20260620 14:55:01.270617 24721 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:01.270689 24721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:01.270841 24721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:01.271972 24721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "1ed1c9768bc445b098586fdbedfab6a9"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "6ee035a113a7e9c65d2ef0269e332f23"
server_key_iv: "9d8f2692f78db1394151433e1b2f0163"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.272392 24721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "1ed1c9768bc445b098586fdbedfab6a9"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "6ee035a113a7e9c65d2ef0269e332f23"
server_key_iv: "9d8f2692f78db1394151433e1b2f0163"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.273811 24721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:55:01.274765 24736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:01.274930 24721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:01.275049 24721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "1ed1c9768bc445b098586fdbedfab6a9"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "6ee035a113a7e9c65d2ef0269e332f23"
server_key_iv: "9d8f2692f78db1394151433e1b2f0163"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.275133 24721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:01.288120 24721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:01.290553 24721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:01.290724 24721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:01.294977 24721 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:32915
I20260620 14:55:01.295017 24788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:32915 every 8 connection(s)
I20260620 14:55:01.295382 24721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:01.296429 24789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:01.297223 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 24721
I20260620 14:55:01.297362 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:01.297575 5594 external_mini_cluster.cc:1428] Setting key 44ca1f8b398dc3ec7704da0cb4190509
I20260620 14:55:01.299739 24789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9: Bootstrap starting.
I20260620 14:55:01.300513 24789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:01.300837 24789 log.cc:824] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:01.301571 24789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9: No bootstrap required, opened a new log
I20260620 14:55:01.303757 24789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed1c9768bc445b098586fdbedfab6a9" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 32915 } }
I20260620 14:55:01.303901 24789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:01.303925 24789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed1c9768bc445b098586fdbedfab6a9, State: Initialized, Role: FOLLOWER
I20260620 14:55:01.304072 24789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed1c9768bc445b098586fdbedfab6a9" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 32915 } }
I20260620 14:55:01.304152 24789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:01.304225 24789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:01.304278 24789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:55:01 dist-test-slave-2m5f krb5kdc[24708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:01.304843 24789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed1c9768bc445b098586fdbedfab6a9" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 32915 } }
I20260620 14:55:01.304975 24789 leader_election.cc:302] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ed1c9768bc445b098586fdbedfab6a9; no voters:
I20260620 14:55:01.305183 24789 leader_election.cc:288] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:01.305222 24794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:01.305343 24794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [term 1 LEADER]: Becoming Leader. State: Replica: 1ed1c9768bc445b098586fdbedfab6a9, State: Running, Role: LEADER
I20260620 14:55:01.305461 24789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:01.305508 24794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed1c9768bc445b098586fdbedfab6a9" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 32915 } }
I20260620 14:55:01.305858 24794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ed1c9768bc445b098586fdbedfab6a9. Latest consensus state: current_term: 1 leader_uuid: "1ed1c9768bc445b098586fdbedfab6a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed1c9768bc445b098586fdbedfab6a9" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 32915 } } }
I20260620 14:55:01.305969 24794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:01.306041 24796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ed1c9768bc445b098586fdbedfab6a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ed1c9768bc445b098586fdbedfab6a9" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 32915 } } }
I20260620 14:55:01.306216 24796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:01.306262 24801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:01.306770 24801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:01.308701 24801 catalog_manager.cc:1347] Generated new cluster ID: 6052f068dc494c378efcac964b613361
I20260620 14:55:01.308758 24801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:01.309311 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.298156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.5.118.190:32915)
0620 14:55:01.298377 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:01.298387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:01.298421 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:55:01.298967 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:01.298985 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:01.299055 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:01.299211 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:01.299828 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.301218 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.302089 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.302718 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.302839 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.305395 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:01.305400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:01.305401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:01.305413 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:01.307465 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:01.307898 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:01.307901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:01.307902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:01.308020 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:01.308286 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:01.308287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:01.308287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:01.308597 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:01.308728 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:01.308956 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:55:01.309100 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
I20260620 14:55:01.320658 24801 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:01.321238 24801 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:01.328474 24801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9: Generated new TSK 0
I20260620 14:55:01.328685 24801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.118.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
Entry for principal kudu/127.5.118.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab.
I20260620 14:55:01.358731 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:32915
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:01.441649 24815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:01.441895 24815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:01.441946 24815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:01.443149 24815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:01.443220 24815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:01.443266 24815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:55:01.444792 24815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.129.keytab
--principal=kudu/127.5.118.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:01.445067 24815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:01.445303 24815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:01.450309 24820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:01.450364 24821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:01.450325 24823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:01.450572 24815 server_base.cc:1034] running on GCE node
I20260620 14:55:01.450807 24815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:01.451071 24815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:01.452239 24815 hybrid_clock.cc:648] HybridClock initialized: now 1781967301452200 us; error 45 us; skew 500 ppm
Jun 20 14:55:01 dist-test-slave-2m5f krb5kdc[24708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967301, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:01.454847 24815 init.cc:377] Logged in from keytab as kudu/127.5.118.129@KRBTEST.COM (short username kudu)
I20260620 14:55:01.455363 24815 webserver.cc:460] Webserver started at http://127.5.118.129:41791/ using document root <none> and password file <none>
I20260620 14:55:01.455616 24815 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:01.455688 24815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:01.455837 24815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:01.456986 24815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "355e4bdb77a4298d6e4797b72addcaf2"
server_key_iv: "aadffe242219e77eeadcdc9387bee426"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.457355 24815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "355e4bdb77a4298d6e4797b72addcaf2"
server_key_iv: "aadffe242219e77eeadcdc9387bee426"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.458688 24815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:55:01.459477 24830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:01.459645 24815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:55:01.459751 24815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "355e4bdb77a4298d6e4797b72addcaf2"
server_key_iv: "aadffe242219e77eeadcdc9387bee426"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.459827 24815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:01.478088 24815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:01.480391 24815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:01.480572 24815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:01.480821 24815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:55:01.481173 24815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:55:01.481230 24815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:01.481263 24815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:55:01.481292 24815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:01.487288 24815 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:40449
I20260620 14:55:01.487371 24943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:40449 every 8 connection(s)
I20260620 14:55:01.487715 24815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
Jun 20 14:55:01 dist-test-slave-2m5f krb5kdc[24708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967301, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.129@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:01.494269 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 24815
I20260620 14:55:01.494402 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:55:01.494679 5594 external_mini_cluster.cc:1428] Setting key 1f7461f15d8e03a7446dbd9d00f7e0d8
I20260620 14:55:01.497869 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.488535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:48321 (local address 127.5.118.190:32915)
0620 14:55:01.488603 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:01.488605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:01.488817 (+ 212us) server_negotiation.cc:407] Connection header received
0620 14:55:01.489410 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:01.489411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:01.489423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:01.489490 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:01.490488 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.490998 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.492049 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.492352 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.492440 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.494608 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:01.494609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:01.494610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:01.494619 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:01.496294 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:01.496856 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:01.496857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:01.496858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:01.496915 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:01.497208 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:01.497208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:01.497209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:01.497315 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:01.497369 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:01.497725 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 14:55:01.497801 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:55:01.498178 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.488524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:32915 (local address 127.5.118.129:48321)
0620 14:55:01.488697 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:01.488734 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:01.489299 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:01.489570 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:01.489573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:01.490177 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:01.490376 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.490381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.491117 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.491117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.491934 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.491937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.492452 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.492453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.492584 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.493065 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:01.493073 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:01.494472 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:01.496393 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:01.496401 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:01.496404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:01.496756 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:01.496988 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:01.496989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:01.496989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:01.497141 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:01.497380 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:01.497386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:01.497582 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 14:55:01.497706 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 14:55:01.497891 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260620 14:55:01.498735 24944 heartbeater.cc:347] Connected to a master server at 127.5.118.190:32915
I20260620 14:55:01.498839 24944 heartbeater.cc:464] Registering TS with master...
I20260620 14:55:01.499074 24944 heartbeater.cc:511] Master 127.5.118.190:32915 requested a full tablet report, sending...
I20260620 14:55:01.499611 24753 ts_manager.cc:194] Registered new tserver with Master: 1fbcc70bf70e4eef8d9b09071a3e1b0f (127.5.118.129:40449)
I20260620 14:55:01.500330 24753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.129@KRBTEST.COM'} at 127.5.118.129:48321
WARNING: no policy specified for kudu/127.5.118.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
Entry for principal kudu/127.5.118.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab.
I20260620 14:55:01.523211 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:32915
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:01.605273 24949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:01.605484 24949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:01.605508 24949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:01.606765 24949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:01.606807 24949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:01.606834 24949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:55:01.608333 24949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.130.keytab
--principal=kudu/127.5.118.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:01.608536 24949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:01.608716 24949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:01.613687 24954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:01.613687 24957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:01.613680 24955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:01.614039 24949 server_base.cc:1034] running on GCE node
I20260620 14:55:01.614281 24949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:01.614539 24949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:01.615708 24949 hybrid_clock.cc:648] HybridClock initialized: now 1781967301615694 us; error 46 us; skew 500 ppm
Jun 20 14:55:01 dist-test-slave-2m5f krb5kdc[24708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967301, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:01.618362 24949 init.cc:377] Logged in from keytab as kudu/127.5.118.130@KRBTEST.COM (short username kudu)
I20260620 14:55:01.618815 24949 webserver.cc:460] Webserver started at http://127.5.118.130:32907/ using document root <none> and password file <none>
I20260620 14:55:01.619071 24949 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:01.619141 24949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:01.619288 24949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:01.620457 24949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "7fb6185834f04e2a8df69cdeaaa73959"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "814abe39867a06a9365d90a9dfb74a50"
server_key_iv: "c4a8f607065f4f126c3a2548ce518502"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.620808 24949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "7fb6185834f04e2a8df69cdeaaa73959"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "814abe39867a06a9365d90a9dfb74a50"
server_key_iv: "c4a8f607065f4f126c3a2548ce518502"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.622131 24949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:55:01.622845 24964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:01.623028 24949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:55:01.623134 24949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "7fb6185834f04e2a8df69cdeaaa73959"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "814abe39867a06a9365d90a9dfb74a50"
server_key_iv: "c4a8f607065f4f126c3a2548ce518502"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.623209 24949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:01.639523 24949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:01.641847 24949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:01.642060 24949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:01.642305 24949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:55:01.642630 24949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:55:01.642688 24949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:01.642728 24949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:55:01.642756 24949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:01.648854 24949 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:40067
I20260620 14:55:01.648937 25077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:40067 every 8 connection(s)
I20260620 14:55:01.649276 24949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
Jun 20 14:55:01 dist-test-slave-2m5f krb5kdc[24708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967301, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.130@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:01.658555 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 24949
I20260620 14:55:01.658732 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:55:01.658957 5594 external_mini_cluster.cc:1428] Setting key ab609413ac502c831c77ba83f59d607a
I20260620 14:55:01.659603 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.650031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:34355 (local address 127.5.118.190:32915)
0620 14:55:01.650099 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:01.650101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:01.650310 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:55:01.650776 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:01.650777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:01.650788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:01.650844 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:01.651787 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.652558 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.653832 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.654055 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.654144 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.656603 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:01.656605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:01.656605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:01.656613 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:01.658052 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:01.658498 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:01.658499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:01.658500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:01.658562 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:01.658847 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:01.658848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:01.658848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:01.658977 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:01.659054 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:01.659416 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 14:55:01.659508 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:55:01.659768 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.650007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:32915 (local address 127.5.118.130:34355)
0620 14:55:01.650184 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:01.650222 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:01.650691 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:01.650926 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:01.650930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:01.651512 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:01.651701 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.651706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.652683 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.652684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.653743 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.653746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.654154 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.654155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.654292 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.654850 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:01.654859 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:01.656501 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:01.658157 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:01.658165 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:01.658168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:01.658397 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:01.658638 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:01.658640 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:01.658640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:01.658789 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:01.659054 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:01.659061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:01.659216 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 14:55:01.659349 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 14:55:01.659508 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260620 14:55:01.660393 25078 heartbeater.cc:347] Connected to a master server at 127.5.118.190:32915
I20260620 14:55:01.660490 25078 heartbeater.cc:464] Registering TS with master...
I20260620 14:55:01.660689 25078 heartbeater.cc:511] Master 127.5.118.190:32915 requested a full tablet report, sending...
I20260620 14:55:01.661083 24753 ts_manager.cc:194] Registered new tserver with Master: 7fb6185834f04e2a8df69cdeaaa73959 (127.5.118.130:40067)
I20260620 14:55:01.661556 24753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.130@KRBTEST.COM'} at 127.5.118.130:34355
WARNING: no policy specified for kudu/127.5.118.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
Entry for principal kudu/127.5.118.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab.
I20260620 14:55:01.685364 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:32915
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:01.768046 25083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:01.768289 25083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:01.768316 25083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:01.769471 25083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:01.769510 25083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:01.769536 25083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:55:01.770961 25083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.131.keytab
--principal=kudu/127.5.118.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:32915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:01.771165 25083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:01.771351 25083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:01.776496 25089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:01.776515 25088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:01.776515 25091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:01.776757 25083 server_base.cc:1034] running on GCE node
I20260620 14:55:01.777007 25083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:01.777223 25083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:01.778376 25083 hybrid_clock.cc:648] HybridClock initialized: now 1781967301778348 us; error 45 us; skew 500 ppm
Jun 20 14:55:01 dist-test-slave-2m5f krb5kdc[24708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967301, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:01.781087 25083 init.cc:377] Logged in from keytab as kudu/127.5.118.131@KRBTEST.COM (short username kudu)
I20260620 14:55:01.781549 25083 webserver.cc:460] Webserver started at http://127.5.118.131:45695/ using document root <none> and password file <none>
I20260620 14:55:01.781786 25083 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:01.781831 25083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:01.781988 25083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:01.783048 25083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "ff83ea8e97c54f3dbf3110bb00a605a6"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "aca61a72b9edc1393db2b2378c25f45a"
server_key_iv: "ab8daa001b7c1c3f555048e1f63f7c2a"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.783428 25083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "ff83ea8e97c54f3dbf3110bb00a605a6"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "aca61a72b9edc1393db2b2378c25f45a"
server_key_iv: "ab8daa001b7c1c3f555048e1f63f7c2a"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.784852 25083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 14:55:01.785627 25098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:01.785807 25083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:55:01.785877 25083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "ff83ea8e97c54f3dbf3110bb00a605a6"
format_stamp: "Formatted at 2026-06-20 14:55:01 on dist-test-slave-2m5f"
server_key: "aca61a72b9edc1393db2b2378c25f45a"
server_key_iv: "ab8daa001b7c1c3f555048e1f63f7c2a"
server_key_version: "encryptionkey@0"
I20260620 14:55:01.785928 25083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:01.812888 25083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:01.815385 25083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:01.815546 25083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:01.815804 25083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:55:01.816146 25083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:55:01.816231 25083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:01.816273 25083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:55:01.816303 25083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:01.822429 25083 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:37281
I20260620 14:55:01.822479 25211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:37281 every 8 connection(s)
I20260620 14:55:01.822809 25083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
Jun 20 14:55:01 dist-test-slave-2m5f krb5kdc[24708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967301, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.131@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:01.830837 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 25083
I20260620 14:55:01.830958 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:55:01.831212 5594 external_mini_cluster.cc:1428] Setting key 868c305893c7eb131798981da60fde70
I20260620 14:55:01.833212 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.823557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:49783 (local address 127.5.118.190:32915)
0620 14:55:01.823622 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:01.823624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:01.823844 (+ 220us) server_negotiation.cc:407] Connection header received
0620 14:55:01.824421 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:01.824423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:01.824433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:01.824504 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:01.825482 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.826018 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.827157 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.827370 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.827451 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.829995 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:01.829996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:01.829996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:01.830003 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:01.831520 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:01.832160 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:01.832161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:01.832161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:01.832242 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:01.832525 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:01.832526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:01.832526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:01.832640 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:01.832694 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:01.833082 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 14:55:01.833147 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:55:01.833492 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.823525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:32915 (local address 127.5.118.131:49783)
0620 14:55:01.823709 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:01.823751 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:01.824312 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:01.824568 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:01.824572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:01.825155 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:01.825380 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.825385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.826137 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.826138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.827044 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.827047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.827458 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.827459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.827606 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.828252 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:01.828262 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:01.829836 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:01.831625 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:01.831634 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:01.831636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:01.832070 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:01.832311 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:01.832312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:01.832312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:01.832448 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:01.832737 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:01.832743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:01.832884 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 14:55:01.833047 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 14:55:01.833186 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":112,"threads_started":1}
I20260620 14:55:01.834086 25212 heartbeater.cc:347] Connected to a master server at 127.5.118.190:32915
I20260620 14:55:01.834193 25212 heartbeater.cc:464] Registering TS with master...
I20260620 14:55:01.834378 25212 heartbeater.cc:511] Master 127.5.118.190:32915 requested a full tablet report, sending...
I20260620 14:55:01.834800 24753 ts_manager.cc:194] Registered new tserver with Master: ff83ea8e97c54f3dbf3110bb00a605a6 (127.5.118.131:37281)
I20260620 14:55:01.835160 24753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.118.131@KRBTEST.COM'} at 127.5.118.131:49783
I20260620 14:55:01.843225 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:55:01.850978 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.844648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.5.118.190:32915)
0620 14:55:01.844726 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:01.844729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:01.844757 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:55:01.844815 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:01.844816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:01.844825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:01.844877 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:01.845506 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.846081 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.846837 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.847047 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.847103 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.848499 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:01.848500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:01.848501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:01.848508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:01.849786 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:01.850160 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:01.850161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:01.850161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:01.850216 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:01.850435 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:01.850436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:01.850436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:01.850598 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:01.850696 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:01.850769 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 14:55:01.850844 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:55:01.852917 24753 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:55:01.854581 24753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:55:01.861861 25223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.857208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:40449 (local address 127.0.0.1:44038)
0620 14:55:01.857494 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:01.857537 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:01.857616 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:01.857915 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:01.857919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:01.857921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:01.858006 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.858008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.859402 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.859403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.860523 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.860525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.861483 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.861484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.861576 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.861599 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:55:01.861709 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:55:01.861773 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":30,"thread_start_us":78,"threads_started":1}
I20260620 14:55:01.861953 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.857312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44038 (local address 127.5.118.129:40449)
0620 14:55:01.857466 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:01.857471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:01.857612 (+ 141us) server_negotiation.cc:407] Connection header received
0620 14:55:01.857705 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:01.857708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:01.857768 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:01.857849 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:01.858099 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.859289 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.860614 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.861389 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.861463 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.861523 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:01.861734 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:55:01.861781 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260620 14:55:01.862303 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.857259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:37281 (local address 127.0.0.1:57836)
0620 14:55:01.857559 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:01.857571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:01.857623 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:01.858258 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:01.858259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:01.858261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:01.858304 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.858306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.859565 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.859566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.860553 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.860555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.862080 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.862081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.862144 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.862148 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:55:01.862199 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 14:55:01.862241 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":93,"threads_started":1}
I20260620 14:55:01.862627 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.857371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.5.118.131:37281)
0620 14:55:01.857987 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:01.857993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:01.858002 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:55:01.858038 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:01.858040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:01.858057 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:01.858185 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:01.858380 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.859442 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.860671 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.861375 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.861459 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.861523 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:01.862502 (+ 979us) server_negotiation.cc:299] Negotiation successful
0620 14:55:01.862548 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":81,"threads_started":1}
I20260620 14:55:01.862661 24878 tablet_service.cc:1467] Processing CreateTablet for tablet a86e1ed5c1a94658bd382a1fd3e83bb5 (DEFAULT_TABLE table=test-table [id=84e782b343e3421293c475ae7587f1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:01.862963 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.857424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:40067 (local address 127.0.0.1:59494)
0620 14:55:01.857812 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:01.857822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:01.857865 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:01.858210 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:01.858211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:01.858213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:01.858269 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.858271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.859119 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.859120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.860523 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.860525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.862603 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.862604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.862662 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.862666 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:55:01.862812 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 14:55:01.862878 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260620 14:55:01.862975 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.857656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.5.118.130:40067)
0620 14:55:01.858045 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:01.858049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:01.858060 (+ 11us) server_negotiation.cc:407] Connection header received
0620 14:55:01.858098 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:01.858101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:01.858116 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:01.858213 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:01.858326 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.859008 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.861565 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.862534 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.862604 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.862667 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:01.862845 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 14:55:01.862894 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":78,"threads_started":1}
I20260620 14:55:01.863062 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86e1ed5c1a94658bd382a1fd3e83bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:01.863322 25146 tablet_service.cc:1467] Processing CreateTablet for tablet a86e1ed5c1a94658bd382a1fd3e83bb5 (DEFAULT_TABLE table=test-table [id=84e782b343e3421293c475ae7587f1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:01.863656 25146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86e1ed5c1a94658bd382a1fd3e83bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:01.864858 25012 tablet_service.cc:1467] Processing CreateTablet for tablet a86e1ed5c1a94658bd382a1fd3e83bb5 (DEFAULT_TABLE table=test-table [id=84e782b343e3421293c475ae7587f1e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:01.865185 25012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86e1ed5c1a94658bd382a1fd3e83bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:01.866410 25229 tablet_bootstrap.cc:492] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f: Bootstrap starting.
I20260620 14:55:01.866902 25230 tablet_bootstrap.cc:492] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6: Bootstrap starting.
I20260620 14:55:01.867260 25229 tablet_bootstrap.cc:654] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:01.867604 25230 tablet_bootstrap.cc:654] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:01.867918 25229 log.cc:824] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:01.868160 25230 log.cc:824] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:01.868834 25229 tablet_bootstrap.cc:492] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f: No bootstrap required, opened a new log
I20260620 14:55:01.868891 25229 ts_tablet_manager.cc:1397] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:55:01.868988 25230 tablet_bootstrap.cc:492] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6: No bootstrap required, opened a new log
I20260620 14:55:01.869071 25230 ts_tablet_manager.cc:1397] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:55:01.870491 25229 raft_consensus.cc:348] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40449 } } peers { permanent_uuid: "7fb6185834f04e2a8df69cdeaaa73959" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40067 } } peers { permanent_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37281 } }
I20260620 14:55:01.870668 25229 raft_consensus.cc:374] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:01.870693 25229 raft_consensus.cc:729] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbcc70bf70e4eef8d9b09071a3e1b0f, State: Initialized, Role: FOLLOWER
I20260620 14:55:01.870685 25230 raft_consensus.cc:348] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40449 } } peers { permanent_uuid: "7fb6185834f04e2a8df69cdeaaa73959" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40067 } } peers { permanent_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37281 } }
I20260620 14:55:01.870831 25229 consensus_queue.cc:260] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40449 } } peers { permanent_uuid: "7fb6185834f04e2a8df69cdeaaa73959" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40067 } } peers { permanent_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37281 } }
I20260620 14:55:01.870873 25230 raft_consensus.cc:374] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:01.870914 25230 raft_consensus.cc:729] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff83ea8e97c54f3dbf3110bb00a605a6, State: Initialized, Role: FOLLOWER
I20260620 14:55:01.871028 25230 consensus_queue.cc:260] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40449 } } peers { permanent_uuid: "7fb6185834f04e2a8df69cdeaaa73959" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40067 } } peers { permanent_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37281 } }
I20260620 14:55:01.871169 25229 ts_tablet_manager.cc:1428] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:55:01.871299 25232 tablet_bootstrap.cc:492] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: Bootstrap starting.
I20260620 14:55:01.871326 25230 ts_tablet_manager.cc:1428] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:55:01.871436 24944 heartbeater.cc:503] Master 127.5.118.190:32915 was elected leader, sending a full tablet report...
I20260620 14:55:01.871670 25212 heartbeater.cc:503] Master 127.5.118.190:32915 was elected leader, sending a full tablet report...
I20260620 14:55:01.872009 25232 tablet_bootstrap.cc:654] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:01.872527 25232 log.cc:824] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:01.873306 25232 tablet_bootstrap.cc:492] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: No bootstrap required, opened a new log
I20260620 14:55:01.873394 25232 ts_tablet_manager.cc:1397] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260620 14:55:01.874909 25232 raft_consensus.cc:348] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40449 } } peers { permanent_uuid: "7fb6185834f04e2a8df69cdeaaa73959" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40067 } } peers { permanent_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37281 } }
I20260620 14:55:01.875083 25232 raft_consensus.cc:374] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:01.875128 25232 raft_consensus.cc:729] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb6185834f04e2a8df69cdeaaa73959, State: Initialized, Role: FOLLOWER
I20260620 14:55:01.875244 25232 consensus_queue.cc:260] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40449 } } peers { permanent_uuid: "7fb6185834f04e2a8df69cdeaaa73959" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40067 } } peers { permanent_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37281 } }
I20260620 14:55:01.875541 25232 ts_tablet_manager.cc:1428] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:55:01.875558 25078 heartbeater.cc:503] Master 127.5.118.190:32915 was elected leader, sending a full tablet report...
W20260620 14:55:01.899854 25079 tablet.cc:2369] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:55:01.924703 25235 raft_consensus.cc:482] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:55:01.924937 25235 raft_consensus.cc:504] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40449 } } peers { permanent_uuid: "7fb6185834f04e2a8df69cdeaaa73959" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40067 } } peers { permanent_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37281 } }
I20260620 14:55:01.925412 25235 leader_election.cc:288] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fbcc70bf70e4eef8d9b09071a3e1b0f (127.5.118.129:40449), 7fb6185834f04e2a8df69cdeaaa73959 (127.5.118.130:40067)
I20260620 14:55:01.928740 25225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.925716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:51699 (local address 127.5.118.129:40449)
0620 14:55:01.925757 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:01.925759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:01.925771 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:55:01.925927 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:01.925928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:01.925939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:01.926022 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:01.926212 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.926857 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.927928 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.928367 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.928429 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.928437 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:01.928616 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 14:55:01.928661 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:55:01.928735 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.925637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:40449 (local address 127.5.118.131:51699)
0620 14:55:01.925714 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:01.925727 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:01.925777 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:01.926053 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:01.926054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:01.926057 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:01.926136 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.926138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.927008 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.927009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.927825 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.927827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.928450 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.928451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.928538 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.928542 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:55:01.928602 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 14:55:01.928639 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":51}
I20260620 14:55:01.928843 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.925863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:43929 (local address 127.5.118.130:40067)
0620 14:55:01.925947 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:01.925949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:01.926042 (+ 93us) server_negotiation.cc:407] Connection header received
0620 14:55:01.926091 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:01.926092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:01.926103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:01.926166 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:01.926350 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.926884 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.927984 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.928461 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.928532 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.928541 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:01.928726 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 14:55:01.928770 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:55:01.928874 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.925769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:40067 (local address 127.5.118.131:43929)
0620 14:55:01.925961 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:01.925971 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:01.926032 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:01.926205 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:01.926206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:01.926208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:01.926279 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.926280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.926989 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.926990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.927852 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:01.927854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.928554 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.928554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:01.928642 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.928646 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:55:01.928702 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:55:01.928738 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":23,"thread_start_us":115,"threads_started":1}
I20260620 14:55:01.929152 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86e1ed5c1a94658bd382a1fd3e83bb5" candidate_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" is_pre_election: true
I20260620 14:55:01.929226 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86e1ed5c1a94658bd382a1fd3e83bb5" candidate_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb6185834f04e2a8df69cdeaaa73959" is_pre_election: true
I20260620 14:55:01.929335 24898 raft_consensus.cc:2454] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff83ea8e97c54f3dbf3110bb00a605a6 in term 0.
I20260620 14:55:01.929373 25032 raft_consensus.cc:2454] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff83ea8e97c54f3dbf3110bb00a605a6 in term 0.
I20260620 14:55:01.929586 25102 leader_election.cc:302] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fbcc70bf70e4eef8d9b09071a3e1b0f, ff83ea8e97c54f3dbf3110bb00a605a6; no voters:
I20260620 14:55:01.929760 25235 raft_consensus.cc:2781] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:55:01.929834 25235 raft_consensus.cc:482] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:55:01.929852 25235 raft_consensus.cc:3037] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:01.930652 25235 raft_consensus.cc:504] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40449 } } peers { permanent_uuid: "7fb6185834f04e2a8df69cdeaaa73959" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40067 } } peers { permanent_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37281 } }
I20260620 14:55:01.930774 25235 leader_election.cc:288] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [CANDIDATE]: Term 1 election: Requested vote from peers 1fbcc70bf70e4eef8d9b09071a3e1b0f (127.5.118.129:40449), 7fb6185834f04e2a8df69cdeaaa73959 (127.5.118.130:40067)
I20260620 14:55:01.931113 24898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86e1ed5c1a94658bd382a1fd3e83bb5" candidate_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f"
I20260620 14:55:01.931135 25032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86e1ed5c1a94658bd382a1fd3e83bb5" candidate_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fb6185834f04e2a8df69cdeaaa73959"
I20260620 14:55:01.931202 25032 raft_consensus.cc:3037] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:01.931224 24898 raft_consensus.cc:3037] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:01.931977 25032 raft_consensus.cc:2454] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff83ea8e97c54f3dbf3110bb00a605a6 in term 1.
I20260620 14:55:01.932006 24898 raft_consensus.cc:2454] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff83ea8e97c54f3dbf3110bb00a605a6 in term 1.
I20260620 14:55:01.932158 25102 leader_election.cc:302] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb6185834f04e2a8df69cdeaaa73959, ff83ea8e97c54f3dbf3110bb00a605a6; no voters:
I20260620 14:55:01.932325 25235 raft_consensus.cc:2781] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:01.932519 25235 raft_consensus.cc:686] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 1 LEADER]: Becoming Leader. State: Replica: ff83ea8e97c54f3dbf3110bb00a605a6, State: Running, Role: LEADER
I20260620 14:55:01.932629 25235 consensus_queue.cc:237] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40449 } } peers { permanent_uuid: "7fb6185834f04e2a8df69cdeaaa73959" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40067 } } peers { permanent_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37281 } }
I20260620 14:55:01.933573 24752 catalog_manager.cc:5495] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff83ea8e97c54f3dbf3110bb00a605a6 (127.5.118.131). New cstate: current_term: 1 leader_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fb6185834f04e2a8df69cdeaaa73959" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff83ea8e97c54f3dbf3110bb00a605a6" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 37281 } health_report { overall_health: HEALTHY } } }
I20260620 14:55:01.956732 25226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.953855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.5.118.131:37281)
0620 14:55:01.953991 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:01.953993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:01.954002 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:55:01.954049 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:01.954050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:01.954069 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:01.954135 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:01.954383 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.954880 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.955701 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:01.955919 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:01.955984 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:01.956281 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:01.956399 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:01.956487 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:01.956567 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:55:01.956589 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 14:55:01.958992 25032 raft_consensus.cc:1261] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959 [term 1 FOLLOWER]: Refusing update from remote peer ff83ea8e97c54f3dbf3110bb00a605a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:55:01.959043 24898 raft_consensus.cc:1261] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 1fbcc70bf70e4eef8d9b09071a3e1b0f [term 1 FOLLOWER]: Refusing update from remote peer ff83ea8e97c54f3dbf3110bb00a605a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:55:01.959504 25235 consensus_queue.cc:1035] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fb6185834f04e2a8df69cdeaaa73959" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 40067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:01.959658 25239 consensus_queue.cc:1035] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fbcc70bf70e4eef8d9b09071a3e1b0f" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 40449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:01.962394 25243 mvcc.cc:205] Tried to move back new op lower bound from 7298938068820414464 to 7298938068716883968. Current Snapshot: MvccSnapshot[applied={T|T < 7298938068820414464}]
I20260620 14:55:01.962558 25244 mvcc.cc:205] Tried to move back new op lower bound from 7298938068820414464 to 7298938068716883968. Current Snapshot: MvccSnapshot[applied={T|T < 7298938068820414464 or (T in {7298938068820414464})}]
I20260620 14:55:01.964404 24753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:55:01.964507 24753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:55:01.965327 24753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ed1c9768bc445b098586fdbedfab6a9: Sending DeleteTablet for 3 replicas of tablet a86e1ed5c1a94658bd382a1fd3e83bb5
I20260620 14:55:01.965669 25146 tablet_service.cc:1514] Processing DeleteTablet for tablet a86e1ed5c1a94658bd382a1fd3e83bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:01 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:57836
I20260620 14:55:01.965708 24878 tablet_service.cc:1514] Processing DeleteTablet for tablet a86e1ed5c1a94658bd382a1fd3e83bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:01 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:44038
I20260620 14:55:01.965956 25252 tablet_replica.cc:330] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6: stopping tablet replica
I20260620 14:55:01.965971 25012 tablet_service.cc:1514] Processing DeleteTablet for tablet a86e1ed5c1a94658bd382a1fd3e83bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:01 UTC) from {username='kudu', principal='kudu/127.5.118.190@KRBTEST.COM'} at 127.0.0.1:59494
I20260620 14:55:01.966056 25252 raft_consensus.cc:2229] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:55:01.966150 25252 raft_consensus.cc:2258] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:55:01.966168 25253 tablet_replica.cc:330] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: stopping tablet replica
I20260620 14:55:01.966256 25253 raft_consensus.cc:2229] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:55:01.966351 25253 raft_consensus.cc:2258] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:55:01.966382 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 24815
I20260620 14:55:01.967159 25252 ts_tablet_manager.cc:1905] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:55:01.967584 25253 ts_tablet_manager.cc:1905] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:55:01.969141 25252 ts_tablet_manager.cc:1918] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:55:01.969210 25252 log.cc:1196] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/a86e1ed5c1a94658bd382a1fd3e83bb5
I20260620 14:55:01.969630 25252 ts_tablet_manager.cc:1939] T a86e1ed5c1a94658bd382a1fd3e83bb5 P ff83ea8e97c54f3dbf3110bb00a605a6: Deleting consensus metadata
I20260620 14:55:01.969689 25253 ts_tablet_manager.cc:1918] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:55:01.969756 25253 log.cc:1196] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/a86e1ed5c1a94658bd382a1fd3e83bb5
I20260620 14:55:01.970033 25253 ts_tablet_manager.cc:1939] T a86e1ed5c1a94658bd382a1fd3e83bb5 P 7fb6185834f04e2a8df69cdeaaa73959: Deleting consensus metadata
I20260620 14:55:01.970625 24740 catalog_manager.cc:4841] TS 7fb6185834f04e2a8df69cdeaaa73959 (127.5.118.130:40067): tablet a86e1ed5c1a94658bd382a1fd3e83bb5 (table test-table [id=84e782b343e3421293c475ae7587f1e6]) successfully deleted
I20260620 14:55:01.972955 24739 catalog_manager.cc:4841] TS ff83ea8e97c54f3dbf3110bb00a605a6 (127.5.118.131:37281): tablet a86e1ed5c1a94658bd382a1fd3e83bb5 (table test-table [id=84e782b343e3421293c475ae7587f1e6]) successfully deleted
W20260620 14:55:01.973804 24740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:40449 (error 108)
I20260620 14:55:01.974053 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 24949
I20260620 14:55:01.975175 25224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:01.975020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:40449 (local address 127.0.0.1:44044)
0620 14:55:01.975085 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:01.975139 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:40449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 14:55:01.975325 24740 catalog_manager.cc:4584] TS 1fbcc70bf70e4eef8d9b09071a3e1b0f (127.5.118.129:40449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a86e1ed5c1a94658bd382a1fd3e83bb5: Network error: Client connection negotiation failed: client connection to 127.5.118.129:40449: connect: Connection refused (error 111)
I20260620 14:55:01.980481 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 25083
I20260620 14:55:01.985262 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 24721
2026-06-20T14:55:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2018 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T14:55:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:02Z Disabled control of system clock
I20260620 14:55:02.013998 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:38631
--webserver_interface=127.5.118.190
--webserver_port=0
--builtin_ntp_servers=127.5.118.148:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:38631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:55:02.095813 25258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:02.096053 25258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:02.096073 25258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:02.097287 25258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:02.097337 25258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:02.097350 25258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:02.097363 25258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:02.097375 25258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:02.098680 25258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:38631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:38631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:02.098891 25258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:02.099131 25258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:02.103936 25266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:02.103945 25263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:02.103945 25264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:02.104429 25258 server_base.cc:1034] running on GCE node
I20260620 14:55:02.104710 25258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:02.105054 25258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:02.106240 25258 hybrid_clock.cc:648] HybridClock initialized: now 1781967302106226 us; error 46 us; skew 500 ppm
I20260620 14:55:02.107648 25258 webserver.cc:460] Webserver started at http://127.5.118.190:33099/ using document root <none> and password file <none>
I20260620 14:55:02.107949 25258 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:02.108035 25258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:02.108247 25258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:02.109390 25258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "bbf84c1b91a5465cbdfbcd4e57e2b2a7"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "7e70eb8721ecd6579af75f5e4e934a1f"
server_key_iv: "ccf56297c854bfc49010f7795697f8bb"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.109763 25258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "bbf84c1b91a5465cbdfbcd4e57e2b2a7"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "7e70eb8721ecd6579af75f5e4e934a1f"
server_key_iv: "ccf56297c854bfc49010f7795697f8bb"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.111274 25258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:55:02.112408 25272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:02.112684 25258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:02.112792 25258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "bbf84c1b91a5465cbdfbcd4e57e2b2a7"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "7e70eb8721ecd6579af75f5e4e934a1f"
server_key_iv: "ccf56297c854bfc49010f7795697f8bb"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.112876 25258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:02.143496 25258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:02.143836 25258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:02.143946 25258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:02.147927 25258 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:38631
I20260620 14:55:02.147984 25324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:38631 every 8 connection(s)
I20260620 14:55:02.148381 25258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:02.149121 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 25258
I20260620 14:55:02.149255 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:02.149537 5594 external_mini_cluster.cc:1428] Setting key 545ac1ad0bc6fc7db0dd757464b96035
I20260620 14:55:02.149659 25325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:02.153081 25325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7: Bootstrap starting.
I20260620 14:55:02.153710 25325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:02.154006 25325 log.cc:824] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:02.155004 25325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7: No bootstrap required, opened a new log
I20260620 14:55:02.155416 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.150123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.5.118.190:38631)
0620 14:55:02.150316 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:02.150323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:02.150350 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:55:02.150861 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:02.150868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:02.150932 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:02.151057 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:02.151292 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.152261 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.153342 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.153841 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.154075 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.154714 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:02.154719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:02.154720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:02.154891 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:02.154923 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:02.154934 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:02.155072 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:02.155147 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:55:02.155230 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260620 14:55:02.157127 25325 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf84c1b91a5465cbdfbcd4e57e2b2a7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38631 } }
I20260620 14:55:02.157334 25325 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:02.157387 25325 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf84c1b91a5465cbdfbcd4e57e2b2a7, State: Initialized, Role: FOLLOWER
I20260620 14:55:02.157509 25325 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf84c1b91a5465cbdfbcd4e57e2b2a7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38631 } }
I20260620 14:55:02.157608 25325 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:02.157652 25325 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:02.157701 25325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:02.158258 25325 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf84c1b91a5465cbdfbcd4e57e2b2a7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38631 } }
I20260620 14:55:02.158377 25325 leader_election.cc:302] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbf84c1b91a5465cbdfbcd4e57e2b2a7; no voters:
I20260620 14:55:02.158568 25325 leader_election.cc:288] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:02.158597 25330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:02.158725 25330 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [term 1 LEADER]: Becoming Leader. State: Replica: bbf84c1b91a5465cbdfbcd4e57e2b2a7, State: Running, Role: LEADER
I20260620 14:55:02.158869 25325 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:02.158867 25330 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf84c1b91a5465cbdfbcd4e57e2b2a7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38631 } }
I20260620 14:55:02.159286 25331 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbf84c1b91a5465cbdfbcd4e57e2b2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf84c1b91a5465cbdfbcd4e57e2b2a7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38631 } } }
I20260620 14:55:02.159302 25332 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbf84c1b91a5465cbdfbcd4e57e2b2a7. Latest consensus state: current_term: 1 leader_uuid: "bbf84c1b91a5465cbdfbcd4e57e2b2a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf84c1b91a5465cbdfbcd4e57e2b2a7" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 38631 } } }
I20260620 14:55:02.159480 25332 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:02.159696 25331 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:02.159933 25339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:02.160468 25339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:02.162177 25339 catalog_manager.cc:1347] Generated new cluster ID: a25018493f464f4c9282d848071a1e3c
I20260620 14:55:02.162237 25339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:02.176762 25339 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:02.177413 25339 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:02.183463 25339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7: Generated new TSK 0
I20260620 14:55:02.183712 25339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:02.189828 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.129:0
--local_ip_for_outbound_sockets=127.5.118.129
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38631
--builtin_ntp_servers=127.5.118.148:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:55:02.277973 25349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:02.278211 25349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:02.278236 25349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:02.279510 25349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:02.279567 25349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:02.279597 25349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.129
I20260620 14:55:02.281114 25349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.118.129
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.129
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:02.281391 25349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:02.281708 25349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:02.288642 25355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:02.288796 25349 server_base.cc:1034] running on GCE node
W20260620 14:55:02.288645 25354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:02.288630 25357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:02.289266 25349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:02.289647 25349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:02.290861 25349 hybrid_clock.cc:648] HybridClock initialized: now 1781967302290829 us; error 50 us; skew 500 ppm
I20260620 14:55:02.292732 25349 webserver.cc:460] Webserver started at http://127.5.118.129:41143/ using document root <none> and password file <none>
I20260620 14:55:02.293152 25349 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:02.293233 25349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:02.293385 25349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:02.294934 25349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/data/instance:
uuid: "ecf62ea3c02a4c1f9b958055024f8928"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "1a86836616e845a4291f769bfba04a43"
server_key_iv: "f84c4e776663c999813ec5ca4fec3d95"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.295478 25349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance:
uuid: "ecf62ea3c02a4c1f9b958055024f8928"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "1a86836616e845a4291f769bfba04a43"
server_key_iv: "f84c4e776663c999813ec5ca4fec3d95"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.297487 25349 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 14:55:02.298564 25363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:02.298794 25349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:02.298892 25349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/wal
uuid: "ecf62ea3c02a4c1f9b958055024f8928"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "1a86836616e845a4291f769bfba04a43"
server_key_iv: "f84c4e776663c999813ec5ca4fec3d95"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.299000 25349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:02.330464 25349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:02.330866 25349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:02.331051 25349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:02.331319 25349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:55:02.331674 25349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:55:02.331730 25349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:02.331765 25349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:55:02.331791 25349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:02.338294 25349 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.129:34909
I20260620 14:55:02.338374 25476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.129:34909 every 8 connection(s)
I20260620 14:55:02.338819 25349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/data/info.pb
I20260620 14:55:02.345273 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.339799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.129:53469 (local address 127.5.118.190:38631)
0620 14:55:02.339862 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:02.339864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:02.340268 (+ 404us) server_negotiation.cc:407] Connection header received
0620 14:55:02.340788 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:02.340790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:02.340802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:02.340886 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:02.341584 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.342137 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.343284 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.343490 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.343560 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.344879 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:02.344881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:02.344881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:02.344888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:02.344986 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:02.344991 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:02.345055 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:02.345164 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:55:02.345198 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:55:02.345525 25479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.339784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38631 (local address 127.5.118.129:53469)
0620 14:55:02.339980 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:02.340018 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:02.340661 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:02.340930 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:02.340934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:02.340944 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:02.341474 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.341479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.342254 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.342255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.343146 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.343149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.343616 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.343618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.343775 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.344689 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:55:02.344689 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:02.344729 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:55:02.344757 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:02.345069 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:02.345080 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:02.345084 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:55:02.345146 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:55:02.345244 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260620 14:55:02.346169 25477 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38631
I20260620 14:55:02.346289 25477 heartbeater.cc:464] Registering TS with master...
I20260620 14:55:02.346387 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 25349
I20260620 14:55:02.346529 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-0/wal/instance
I20260620 14:55:02.346561 25477 heartbeater.cc:511] Master 127.5.118.190:38631 requested a full tablet report, sending...
I20260620 14:55:02.346761 5594 external_mini_cluster.cc:1428] Setting key 30aca94c3cc26f8e03355cb1d18a6069
I20260620 14:55:02.347273 25289 ts_manager.cc:194] Registered new tserver with Master: ecf62ea3c02a4c1f9b958055024f8928 (127.5.118.129:34909)
I20260620 14:55:02.347945 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.130:0
--local_ip_for_outbound_sockets=127.5.118.130
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38631
--builtin_ntp_servers=127.5.118.148:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:55:02.348284 25289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.129:53469
W20260620 14:55:02.435701 25480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:02.435937 25480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:02.435974 25480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:02.437264 25480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:02.437317 25480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:02.437345 25480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.130
I20260620 14:55:02.438674 25480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.118.130
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.130
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:02.438879 25480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:02.439152 25480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:02.445010 25485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:02.445019 25486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:02.445070 25488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:02.445063 25480 server_base.cc:1034] running on GCE node
I20260620 14:55:02.445529 25480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:02.445824 25480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:02.447032 25480 hybrid_clock.cc:648] HybridClock initialized: now 1781967302447031 us; error 40 us; skew 500 ppm
I20260620 14:55:02.448540 25480 webserver.cc:460] Webserver started at http://127.5.118.130:33597/ using document root <none> and password file <none>
I20260620 14:55:02.448812 25480 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:02.448884 25480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:02.449059 25480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:02.450518 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/data/instance:
uuid: "8af9dfed76164f52a6ddf5f9f63ac259"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "c680ca427f3c98f908ec6428e74949b2"
server_key_iv: "99ab3f5bae8a9ebe649ff0728827dd52"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.450932 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance:
uuid: "8af9dfed76164f52a6ddf5f9f63ac259"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "c680ca427f3c98f908ec6428e74949b2"
server_key_iv: "99ab3f5bae8a9ebe649ff0728827dd52"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.452587 25480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 14:55:02.453601 25494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:02.453792 25480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:02.453907 25480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/wal
uuid: "8af9dfed76164f52a6ddf5f9f63ac259"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "c680ca427f3c98f908ec6428e74949b2"
server_key_iv: "99ab3f5bae8a9ebe649ff0728827dd52"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.454071 25480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:02.481040 25480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:02.481433 25480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:02.481609 25480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:02.481873 25480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:55:02.482245 25480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:55:02.482306 25480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:02.482349 25480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:55:02.482374 25480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:02.488775 25480 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.130:34943
I20260620 14:55:02.488822 25607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.130:34943 every 8 connection(s)
I20260620 14:55:02.489248 25480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/data/info.pb
I20260620 14:55:02.493690 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 25480
I20260620 14:55:02.493791 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/wal/instance
I20260620 14:55:02.494032 5594 external_mini_cluster.cc:1428] Setting key ecaae0685516b2d322c64e02cd636398
I20260620 14:55:02.495182 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.118.131:0
--local_ip_for_outbound_sockets=127.5.118.131
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38631
--builtin_ntp_servers=127.5.118.148:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:55:02.495432 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.490071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:37861 (local address 127.5.118.190:38631)
0620 14:55:02.490141 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:02.490143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:02.490418 (+ 275us) server_negotiation.cc:407] Connection header received
0620 14:55:02.491028 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:02.491030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:02.491046 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:02.491094 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:02.491839 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.492487 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.493618 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.493887 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.493987 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.495107 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:02.495109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:02.495109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:02.495114 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:02.495131 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:02.495135 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:02.495189 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:02.495312 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:55:02.495348 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:55:02.495909 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.490072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38631 (local address 127.5.118.130:37861)
0620 14:55:02.490285 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:02.490326 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:02.490897 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:02.491172 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:02.491176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:02.491186 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:02.491717 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.491722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.492609 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.492610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.493505 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.493508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.494003 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.494004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.494200 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.494922 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:55:02.494922 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:02.494964 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:55:02.494993 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:02.495201 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:02.495206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:02.495209 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:55:02.495613 (+ 404us) client_negotiation.cc:240] Negotiation successful
0620 14:55:02.495730 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260620 14:55:02.496534 25608 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38631
I20260620 14:55:02.496670 25608 heartbeater.cc:464] Registering TS with master...
I20260620 14:55:02.496901 25608 heartbeater.cc:511] Master 127.5.118.190:38631 requested a full tablet report, sending...
I20260620 14:55:02.497361 25289 ts_manager.cc:194] Registered new tserver with Master: 8af9dfed76164f52a6ddf5f9f63ac259 (127.5.118.130:34943)
I20260620 14:55:02.497918 25289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.130:37861
W20260620 14:55:02.581645 25611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:02.581936 25611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:02.581975 25611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:02.583256 25611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:02.583346 25611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:02.583391 25611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.118.131
I20260620 14:55:02.584939 25611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.118.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.118.131
--webserver_port=0
--tserver_master_addrs=127.5.118.190:38631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.118.131
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:02.585260 25611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:02.585614 25611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:02.591457 25619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:02.591450 25617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:02.591444 25616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:02.591523 25611 server_base.cc:1034] running on GCE node
I20260620 14:55:02.592018 25611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:02.592357 25611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:02.593544 25611 hybrid_clock.cc:648] HybridClock initialized: now 1781967302593521 us; error 60 us; skew 500 ppm
I20260620 14:55:02.595227 25611 webserver.cc:460] Webserver started at http://127.5.118.131:44431/ using document root <none> and password file <none>
I20260620 14:55:02.595528 25611 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:02.595610 25611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:02.595803 25611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:02.597110 25611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/data/instance:
uuid: "56c41e82bded418aa5a4888bb1f5ca55"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "771d8baf7031e06528a2e64c09adab99"
server_key_iv: "d68c92d65062bf188029a909c99637ac"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.597505 25611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance:
uuid: "56c41e82bded418aa5a4888bb1f5ca55"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "771d8baf7031e06528a2e64c09adab99"
server_key_iv: "d68c92d65062bf188029a909c99637ac"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.599323 25611 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 14:55:02.600449 25625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:02.600701 25611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:55:02.600826 25611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/wal
uuid: "56c41e82bded418aa5a4888bb1f5ca55"
format_stamp: "Formatted at 2026-06-20 14:55:02 on dist-test-slave-2m5f"
server_key: "771d8baf7031e06528a2e64c09adab99"
server_key_iv: "d68c92d65062bf188029a909c99637ac"
server_key_version: "encryptionkey@0"
I20260620 14:55:02.600929 25611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:02.623975 25611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:02.624372 25611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:02.624539 25611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:02.624792 25611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:55:02.625160 25611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:55:02.625219 25611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:02.625262 25611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:55:02.625295 25611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:02.631309 25611 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.131:33675
I20260620 14:55:02.631367 25738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.131:33675 every 8 connection(s)
I20260620 14:55:02.631757 25611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/data/info.pb
I20260620 14:55:02.637571 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.632650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.131:56945 (local address 127.5.118.190:38631)
0620 14:55:02.632726 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:02.632728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:02.632989 (+ 261us) server_negotiation.cc:407] Connection header received
0620 14:55:02.633631 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:02.633633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:02.633645 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:02.633703 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:02.634396 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.634915 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.635933 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.636140 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.636285 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.637261 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:02.637262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:02.637263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:02.637268 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:02.637287 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:02.637296 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:02.637359 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:02.637456 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 14:55:02.637486 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:55:02.637838 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.632639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:38631 (local address 127.5.118.131:56945)
0620 14:55:02.632865 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:02.632900 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:02.633463 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:02.633784 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:02.633788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:02.633796 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:02.634298 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.634302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.635020 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.635021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.635827 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.635829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.636298 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.636299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.636439 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.637098 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:55:02.637098 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:02.637130 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:55:02.637154 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:02.637358 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:02.637365 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:02.637368 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:55:02.637459 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:55:02.637576 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":103,"threads_started":1}
I20260620 14:55:02.638478 25739 heartbeater.cc:347] Connected to a master server at 127.5.118.190:38631
I20260620 14:55:02.638568 25739 heartbeater.cc:464] Registering TS with master...
I20260620 14:55:02.638715 25739 heartbeater.cc:511] Master 127.5.118.190:38631 requested a full tablet report, sending...
I20260620 14:55:02.639097 25289 ts_manager.cc:194] Registered new tserver with Master: 56c41e82bded418aa5a4888bb1f5ca55 (127.5.118.131:33675)
I20260620 14:55:02.639421 25289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.118.131:56945
I20260620 14:55:02.641990 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 25611
I20260620 14:55:02.642089 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/wal/instance
I20260620 14:55:02.642293 5594 external_mini_cluster.cc:1428] Setting key 5d37a1855a1bca4f0288cc66238781b3
I20260620 14:55:02.643420 5594 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:55:02.648316 25328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.644862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36904 (local address 127.5.118.190:38631)
0620 14:55:02.644925 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:02.644927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:02.645260 (+ 333us) server_negotiation.cc:407] Connection header received
0620 14:55:02.645411 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:02.645411 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:02.645429 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:02.645496 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:02.645682 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.646171 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.647024 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.647238 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.647323 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.648060 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:02.648062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:02.648062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:02.648067 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:02.648082 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:02.648086 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:02.648134 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:02.648201 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:55:02.648224 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:55:02.650262 25289 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:55:02.651798 25289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:55:02.659392 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.654363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:34909 (local address 127.0.0.1:50196)
0620 14:55:02.654514 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:02.654552 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:02.654615 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:02.655122 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:02.655125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:02.655128 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:02.655205 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.655206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.656786 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.656787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.658074 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.658077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.659048 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.659049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.659157 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.659173 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:55:02.659242 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:55:02.659299 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":19,"thread_start_us":102,"threads_started":1}
I20260620 14:55:02.659435 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.654729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.5.118.129:34909)
0620 14:55:02.654930 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:02.654936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:02.654946 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:55:02.654990 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:02.654993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:02.655009 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:02.655454 (+ 445us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:02.655522 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.656691 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.658194 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.658956 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.659030 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.659123 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:02.659286 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:55:02.659330 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260620 14:55:02.659688 25750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.654585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.130:34943 (local address 127.0.0.1:34718)
0620 14:55:02.654794 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:02.654801 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:02.654866 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:02.655197 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:02.655198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:02.655200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:02.655290 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.655293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.656565 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.656566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.658187 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.658191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.659421 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.659422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.659487 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.659502 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 14:55:02.659564 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:55:02.659613 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260620 14:55:02.659932 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.654806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.5.118.130:34943)
0620 14:55:02.654967 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:02.654972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:02.654982 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:55:02.655033 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:02.655035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:02.655052 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:02.655130 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:02.655350 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.656454 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.658284 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.659079 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.659149 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.659241 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:02.659796 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 14:55:02.659850 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":104,"threads_started":1}
I20260620 14:55:02.660141 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.654363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:33675 (local address 127.0.0.1:55758)
0620 14:55:02.654473 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:02.654553 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:02.654616 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:02.655201 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:02.655202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:02.655204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:02.655275 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.655276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.656903 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.656904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.658074 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.658077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.659868 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.659869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.659966 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.659971 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:55:02.660039 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:55:02.660080 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":35,"thread_start_us":63,"threads_started":1}
I20260620 14:55:02.660444 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.654538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55758 (local address 127.5.118.131:33675)
0620 14:55:02.655000 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:02.655004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:02.655013 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:55:02.655049 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:02.655051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:02.655066 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:02.655135 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:02.655623 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.656824 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.658170 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.658952 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.659030 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.659123 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:02.660317 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0620 14:55:02.660365 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":94,"threads_started":1}
I20260620 14:55:02.660522 25411 tablet_service.cc:1467] Processing CreateTablet for tablet 141e06aa8a084c2fbdf26023fe26cd25 (DEFAULT_TABLE table=test-table [id=6551664eb6ea460d8b8c6329657be4c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:02.660535 25542 tablet_service.cc:1467] Processing CreateTablet for tablet 141e06aa8a084c2fbdf26023fe26cd25 (DEFAULT_TABLE table=test-table [id=6551664eb6ea460d8b8c6329657be4c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:02.660895 25542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141e06aa8a084c2fbdf26023fe26cd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:02.660910 25411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141e06aa8a084c2fbdf26023fe26cd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:02.661226 25673 tablet_service.cc:1467] Processing CreateTablet for tablet 141e06aa8a084c2fbdf26023fe26cd25 (DEFAULT_TABLE table=test-table [id=6551664eb6ea460d8b8c6329657be4c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:55:02.661465 25673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141e06aa8a084c2fbdf26023fe26cd25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:02.663925 25753 tablet_bootstrap.cc:492] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55: Bootstrap starting.
I20260620 14:55:02.664438 25755 tablet_bootstrap.cc:492] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928: Bootstrap starting.
I20260620 14:55:02.664780 25753 tablet_bootstrap.cc:654] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:02.665117 25753 log.cc:824] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:02.665150 25755 tablet_bootstrap.cc:654] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:02.665464 25755 log.cc:824] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:02.665812 25754 tablet_bootstrap.cc:492] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259: Bootstrap starting.
I20260620 14:55:02.666007 25753 tablet_bootstrap.cc:492] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55: No bootstrap required, opened a new log
I20260620 14:55:02.666086 25753 ts_tablet_manager.cc:1397] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:55:02.666216 25755 tablet_bootstrap.cc:492] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928: No bootstrap required, opened a new log
I20260620 14:55:02.666304 25755 ts_tablet_manager.cc:1397] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:55:02.666523 25754 tablet_bootstrap.cc:654] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:02.666855 25754 log.cc:824] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:02.667694 25754 tablet_bootstrap.cc:492] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259: No bootstrap required, opened a new log
I20260620 14:55:02.667765 25754 ts_tablet_manager.cc:1397] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:55:02.668658 25755 raft_consensus.cc:348] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c41e82bded418aa5a4888bb1f5ca55" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33675 } } peers { permanent_uuid: "ecf62ea3c02a4c1f9b958055024f8928" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 34909 } } peers { permanent_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34943 } }
I20260620 14:55:02.668824 25755 raft_consensus.cc:374] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:02.668886 25755 raft_consensus.cc:729] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf62ea3c02a4c1f9b958055024f8928, State: Initialized, Role: FOLLOWER
I20260620 14:55:02.668892 25753 raft_consensus.cc:348] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c41e82bded418aa5a4888bb1f5ca55" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33675 } } peers { permanent_uuid: "ecf62ea3c02a4c1f9b958055024f8928" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 34909 } } peers { permanent_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34943 } }
I20260620 14:55:02.669061 25753 raft_consensus.cc:374] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:02.669031 25755 consensus_queue.cc:260] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c41e82bded418aa5a4888bb1f5ca55" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33675 } } peers { permanent_uuid: "ecf62ea3c02a4c1f9b958055024f8928" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 34909 } } peers { permanent_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34943 } }
I20260620 14:55:02.669149 25753 raft_consensus.cc:729] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c41e82bded418aa5a4888bb1f5ca55, State: Initialized, Role: FOLLOWER
I20260620 14:55:02.669305 25755 ts_tablet_manager.cc:1428] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:55:02.669312 25753 consensus_queue.cc:260] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c41e82bded418aa5a4888bb1f5ca55" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33675 } } peers { permanent_uuid: "ecf62ea3c02a4c1f9b958055024f8928" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 34909 } } peers { permanent_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34943 } }
I20260620 14:55:02.669348 25477 heartbeater.cc:503] Master 127.5.118.190:38631 was elected leader, sending a full tablet report...
I20260620 14:55:02.669652 25753 ts_tablet_manager.cc:1428] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260620 14:55:02.669696 25739 heartbeater.cc:503] Master 127.5.118.190:38631 was elected leader, sending a full tablet report...
I20260620 14:55:02.670480 25754 raft_consensus.cc:348] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c41e82bded418aa5a4888bb1f5ca55" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33675 } } peers { permanent_uuid: "ecf62ea3c02a4c1f9b958055024f8928" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 34909 } } peers { permanent_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34943 } }
I20260620 14:55:02.670670 25754 raft_consensus.cc:374] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:02.670719 25754 raft_consensus.cc:729] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8af9dfed76164f52a6ddf5f9f63ac259, State: Initialized, Role: FOLLOWER
I20260620 14:55:02.670830 25754 consensus_queue.cc:260] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c41e82bded418aa5a4888bb1f5ca55" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33675 } } peers { permanent_uuid: "ecf62ea3c02a4c1f9b958055024f8928" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 34909 } } peers { permanent_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34943 } }
I20260620 14:55:02.671085 25754 ts_tablet_manager.cc:1428] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:55:02.671152 25608 heartbeater.cc:503] Master 127.5.118.190:38631 was elected leader, sending a full tablet report...
I20260620 14:55:02.695551 25761 raft_consensus.cc:482] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:55:02.695711 25761 raft_consensus.cc:504] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c41e82bded418aa5a4888bb1f5ca55" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33675 } } peers { permanent_uuid: "ecf62ea3c02a4c1f9b958055024f8928" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 34909 } } peers { permanent_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34943 } }
I20260620 14:55:02.696127 25761 leader_election.cc:288] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56c41e82bded418aa5a4888bb1f5ca55 (127.5.118.131:33675), ecf62ea3c02a4c1f9b958055024f8928 (127.5.118.129:34909)
I20260620 14:55:02.699146 25751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.696511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:35557 (local address 127.5.118.129:34909)
0620 14:55:02.696569 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:02.696571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:02.696580 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:55:02.696608 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:02.696609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:02.696624 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:02.696679 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:02.696863 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.697366 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.698370 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.698783 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.698840 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.698847 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:02.699018 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 14:55:02.699071 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 14:55:02.699172 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.696339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:34909 (local address 127.5.118.130:35557)
0620 14:55:02.696416 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:02.696429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:02.696471 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:02.696699 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:02.696700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:02.696703 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:02.696779 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.696781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.697483 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.697484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.698275 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.698277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.698864 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.698865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.698940 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.698943 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:55:02.698996 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:55:02.699033 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:55:02.699204 25749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.696448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.118.130:41089 (local address 127.5.118.131:33675)
0620 14:55:02.696550 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:02.696552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:02.696618 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:55:02.696685 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:02.696686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:02.696696 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:02.696769 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:55:02.696951 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.697471 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.698454 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.698811 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.698906 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.698914 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:02.699091 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 14:55:02.699135 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:55:02.699328 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.696339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.131:33675 (local address 127.5.118.130:41089)
0620 14:55:02.696544 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:02.696554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:02.696609 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:02.696784 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:02.696784 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:02.696786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:55:02.696868 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.696870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.697557 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.697557 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.698365 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:02.698368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.698929 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.698930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:02.699011 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.699015 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 14:55:02.699071 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:55:02.699105 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":12,"thread_start_us":130,"threads_started":1}
I20260620 14:55:02.699596 25431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141e06aa8a084c2fbdf26023fe26cd25" candidate_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf62ea3c02a4c1f9b958055024f8928" is_pre_election: true
I20260620 14:55:02.699638 25693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141e06aa8a084c2fbdf26023fe26cd25" candidate_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c41e82bded418aa5a4888bb1f5ca55" is_pre_election: true
I20260620 14:55:02.699767 25431 raft_consensus.cc:2454] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8af9dfed76164f52a6ddf5f9f63ac259 in term 0.
I20260620 14:55:02.699767 25693 raft_consensus.cc:2454] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8af9dfed76164f52a6ddf5f9f63ac259 in term 0.
I20260620 14:55:02.700011 25495 leader_election.cc:302] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c41e82bded418aa5a4888bb1f5ca55, 8af9dfed76164f52a6ddf5f9f63ac259; no voters:
I20260620 14:55:02.700155 25761 raft_consensus.cc:2781] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:55:02.700244 25761 raft_consensus.cc:482] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:55:02.700269 25761 raft_consensus.cc:3037] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:02.700999 25761 raft_consensus.cc:504] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c41e82bded418aa5a4888bb1f5ca55" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33675 } } peers { permanent_uuid: "ecf62ea3c02a4c1f9b958055024f8928" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 34909 } } peers { permanent_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34943 } }
I20260620 14:55:02.701164 25761 leader_election.cc:288] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [CANDIDATE]: Term 1 election: Requested vote from peers 56c41e82bded418aa5a4888bb1f5ca55 (127.5.118.131:33675), ecf62ea3c02a4c1f9b958055024f8928 (127.5.118.129:34909)
I20260620 14:55:02.701297 25693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141e06aa8a084c2fbdf26023fe26cd25" candidate_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c41e82bded418aa5a4888bb1f5ca55"
I20260620 14:55:02.701344 25431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141e06aa8a084c2fbdf26023fe26cd25" candidate_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf62ea3c02a4c1f9b958055024f8928"
I20260620 14:55:02.701406 25693 raft_consensus.cc:3037] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:02.701450 25431 raft_consensus.cc:3037] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:02.702345 25693 raft_consensus.cc:2454] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8af9dfed76164f52a6ddf5f9f63ac259 in term 1.
I20260620 14:55:02.702517 25495 leader_election.cc:302] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c41e82bded418aa5a4888bb1f5ca55, 8af9dfed76164f52a6ddf5f9f63ac259; no voters:
I20260620 14:55:02.702600 25761 raft_consensus.cc:2781] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:02.702639 25431 raft_consensus.cc:2454] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8af9dfed76164f52a6ddf5f9f63ac259 in term 1.
I20260620 14:55:02.702826 25761 raft_consensus.cc:686] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 1 LEADER]: Becoming Leader. State: Replica: 8af9dfed76164f52a6ddf5f9f63ac259, State: Running, Role: LEADER
I20260620 14:55:02.702929 25761 consensus_queue.cc:237] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c41e82bded418aa5a4888bb1f5ca55" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33675 } } peers { permanent_uuid: "ecf62ea3c02a4c1f9b958055024f8928" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 34909 } } peers { permanent_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34943 } }
I20260620 14:55:02.703831 25289 catalog_manager.cc:5495] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8af9dfed76164f52a6ddf5f9f63ac259 (127.5.118.130). New cstate: current_term: 1 leader_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c41e82bded418aa5a4888bb1f5ca55" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecf62ea3c02a4c1f9b958055024f8928" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 34909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8af9dfed76164f52a6ddf5f9f63ac259" member_type: VOTER last_known_addr { host: "127.5.118.130" port: 34943 } health_report { overall_health: HEALTHY } } }
I20260620 14:55:02.719568 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.716626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.5.118.130:34943)
0620 14:55:02.716691 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:02.716693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:02.716709 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:55:02.716747 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:02.716747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:02.716768 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:02.716834 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:02.717051 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.717580 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.718474 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:02.718691 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:02.718748 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:02.719087 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:02.719213 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:02.719296 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:02.719397 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:55:02.719419 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:55:02.721776 25693 raft_consensus.cc:1261] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55 [term 1 FOLLOWER]: Refusing update from remote peer 8af9dfed76164f52a6ddf5f9f63ac259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:55:02.721776 25431 raft_consensus.cc:1261] T 141e06aa8a084c2fbdf26023fe26cd25 P ecf62ea3c02a4c1f9b958055024f8928 [term 1 FOLLOWER]: Refusing update from remote peer 8af9dfed76164f52a6ddf5f9f63ac259: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:55:02.722178 25761 consensus_queue.cc:1035] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf62ea3c02a4c1f9b958055024f8928" member_type: VOTER last_known_addr { host: "127.5.118.129" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:02.722329 25763 consensus_queue.cc:1035] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c41e82bded418aa5a4888bb1f5ca55" member_type: VOTER last_known_addr { host: "127.5.118.131" port: 33675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:02.724593 25766 mvcc.cc:205] Tried to move back new op lower bound from 7298938071944871936 to 7298938071871856640. Current Snapshot: MvccSnapshot[applied={T|T < 7298938071944871936}]
I20260620 14:55:02.725560 25767 mvcc.cc:205] Tried to move back new op lower bound from 7298938071944871936 to 7298938071871856640. Current Snapshot: MvccSnapshot[applied={T|T < 7298938071944871936}]
I20260620 14:55:02.725841 25768 mvcc.cc:205] Tried to move back new op lower bound from 7298938071944871936 to 7298938071871856640. Current Snapshot: MvccSnapshot[applied={T|T < 7298938071944871936 or (T in {7298938071944871936})}]
I20260620 14:55:02.727057 25289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:55:02.727156 25289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:55:02.727890 25289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbf84c1b91a5465cbdfbcd4e57e2b2a7: Sending DeleteTablet for 3 replicas of tablet 141e06aa8a084c2fbdf26023fe26cd25
I20260620 14:55:02.728264 25542 tablet_service.cc:1514] Processing DeleteTablet for tablet 141e06aa8a084c2fbdf26023fe26cd25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:02 UTC) from {username='slave'} at 127.0.0.1:34718
I20260620 14:55:02.728278 25411 tablet_service.cc:1514] Processing DeleteTablet for tablet 141e06aa8a084c2fbdf26023fe26cd25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:02 UTC) from {username='slave'} at 127.0.0.1:50196
I20260620 14:55:02.728343 25673 tablet_service.cc:1514] Processing DeleteTablet for tablet 141e06aa8a084c2fbdf26023fe26cd25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:55:02 UTC) from {username='slave'} at 127.0.0.1:55758
I20260620 14:55:02.728531 25777 tablet_replica.cc:330] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259: stopping tablet replica
I20260620 14:55:02.728613 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 25349
I20260620 14:55:02.728703 25777 raft_consensus.cc:2229] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:55:02.728891 25777 raft_consensus.cc:2258] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:55:02.729466 25777 ts_tablet_manager.cc:1905] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:55:02.731642 25777 ts_tablet_manager.cc:1918] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:55:02.731798 25777 log.cc:1196] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-1/wal/wals/141e06aa8a084c2fbdf26023fe26cd25
I20260620 14:55:02.732208 25777 ts_tablet_manager.cc:1939] T 141e06aa8a084c2fbdf26023fe26cd25 P 8af9dfed76164f52a6ddf5f9f63ac259: Deleting consensus metadata
I20260620 14:55:02.732343 25778 tablet_replica.cc:330] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55: stopping tablet replica
I20260620 14:55:02.732537 25778 raft_consensus.cc:2229] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:55:02.732731 25778 raft_consensus.cc:2258] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:55:02.732774 25274 catalog_manager.cc:4841] TS 8af9dfed76164f52a6ddf5f9f63ac259 (127.5.118.130:34943): tablet 141e06aa8a084c2fbdf26023fe26cd25 (table test-table [id=6551664eb6ea460d8b8c6329657be4c9]) successfully deleted
I20260620 14:55:02.733366 25778 ts_tablet_manager.cc:1905] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:55:02.735190 25778 ts_tablet_manager.cc:1918] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:55:02.735275 25778 log.cc:1196] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55: Deleting WAL directory at /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967053866185-5594-0/minicluster-data/ts-2/wal/wals/141e06aa8a084c2fbdf26023fe26cd25
I20260620 14:55:02.735541 25778 ts_tablet_manager.cc:1939] T 141e06aa8a084c2fbdf26023fe26cd25 P 56c41e82bded418aa5a4888bb1f5ca55: Deleting consensus metadata
I20260620 14:55:02.736057 25273 catalog_manager.cc:4841] TS 56c41e82bded418aa5a4888bb1f5ca55 (127.5.118.131:33675): tablet 141e06aa8a084c2fbdf26023fe26cd25 (table test-table [id=6551664eb6ea460d8b8c6329657be4c9]) successfully deleted
W20260620 14:55:02.736702 25274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.118.129:34909 (error 108)
I20260620 14:55:02.737259 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:02.737113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.129:34909 (local address 127.0.0.1:50204)
0620 14:55:02.737177 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:02.737225 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.129:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 14:55:02.737291 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 25480
W20260620 14:55:02.738200 25274 catalog_manager.cc:4584] TS ecf62ea3c02a4c1f9b958055024f8928 (127.5.118.129:34909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 141e06aa8a084c2fbdf26023fe26cd25: Network error: Client connection negotiation failed: client connection to 127.5.118.129:34909: connect: Connection refused (error 111)
I20260620 14:55:02.743338 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 25611
I20260620 14:55:02.748319 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 25258
2026-06-20T14:55:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (762 ms)
[----------] 6 tests from EncryptionPolicyTest (8880 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:02 dist-test-slave-2m5f krb5kdc[25782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:02 dist-test-slave-2m5f krb5kdc[25782](info): set up 2 sockets
Jun 20 14:55:02 dist-test-slave-2m5f krb5kdc[25782](info): commencing operation
krb5kdc: starting...
W20260620 14:55:03.796761 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:03 dist-test-slave-2m5f krb5kdc[25782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:03.920552 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:04.003870 25795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:04.004137 25795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:04.004180 25795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:04.005381 25795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:04.005460 25795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:04.005482 25795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:04.005503 25795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:04.005533 25795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:04.007048 25795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39293
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:04.007346 25795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:04.007606 25795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:04.013006 25800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:04.012990 25803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:04.012990 25801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:04.013171 25795 server_base.cc:1034] running on GCE node
I20260620 14:55:04.013574 25795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:04.013878 25795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:04.015232 25795 hybrid_clock.cc:648] HybridClock initialized: now 1781967304015221 us; error 37 us; skew 500 ppm
Jun 20 14:55:04 dist-test-slave-2m5f krb5kdc[25782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:04.018193 25795 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:04.018736 25795 webserver.cc:460] Webserver started at http://127.0.0.1:33729/ using document root <none> and password file <none>
I20260620 14:55:04.019004 25795 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:04.019078 25795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:04.019229 25795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:04.020443 25795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "f4f3ccae2b53461e9f2936cf789c2b5d"
format_stamp: "Formatted at 2026-06-20 14:55:04 on dist-test-slave-2m5f"
server_key: "57927c2608edd5c855b200e25ab6af5e"
server_key_iv: "aa50540fce07025e3051f39a7cf4eaf5"
server_key_version: "encryptionkey@0"
I20260620 14:55:04.020812 25795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "f4f3ccae2b53461e9f2936cf789c2b5d"
format_stamp: "Formatted at 2026-06-20 14:55:04 on dist-test-slave-2m5f"
server_key: "57927c2608edd5c855b200e25ab6af5e"
server_key_iv: "aa50540fce07025e3051f39a7cf4eaf5"
server_key_version: "encryptionkey@0"
I20260620 14:55:04.022253 25795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:55:04.023118 25810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:04.023384 25795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:04.023514 25795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "f4f3ccae2b53461e9f2936cf789c2b5d"
format_stamp: "Formatted at 2026-06-20 14:55:04 on dist-test-slave-2m5f"
server_key: "57927c2608edd5c855b200e25ab6af5e"
server_key_iv: "aa50540fce07025e3051f39a7cf4eaf5"
server_key_version: "encryptionkey@0"
I20260620 14:55:04.023607 25795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:04.047150 25795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:04.049376 25795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:04.049592 25795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:04.054018 25795 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39293
I20260620 14:55:04.054056 25862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39293 every 8 connection(s)
I20260620 14:55:04.054466 25795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:04.055572 25863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:04.056016 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 25795
I20260620 14:55:04.056154 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:04.056452 5594 external_mini_cluster.cc:1428] Setting key 7db8560c22c7ffe27f982ac8709c8574
I20260620 14:55:04.059015 25863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d: Bootstrap starting.
I20260620 14:55:04.059772 25863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:04.060217 25863 log.cc:824] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:04.061094 25863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d: No bootstrap required, opened a new log
I20260620 14:55:04.063540 25863 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3ccae2b53461e9f2936cf789c2b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } }
I20260620 14:55:04.063702 25863 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:04.063727 25863 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f3ccae2b53461e9f2936cf789c2b5d, State: Initialized, Role: FOLLOWER
I20260620 14:55:04.063885 25863 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3ccae2b53461e9f2936cf789c2b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } }
I20260620 14:55:04.063978 25863 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:04.064004 25863 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:04.064090 25863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [term 0 FOLLOWER]: Advancing to term 1
Jun 20 14:55:04 dist-test-slave-2m5f krb5kdc[25782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:55:04.064814 25863 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3ccae2b53461e9f2936cf789c2b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } }
I20260620 14:55:04.064913 25863 leader_election.cc:302] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4f3ccae2b53461e9f2936cf789c2b5d; no voters:
I20260620 14:55:04.065199 25863 leader_election.cc:288] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:04.065232 25868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:04.065359 25868 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [term 1 LEADER]: Becoming Leader. State: Replica: f4f3ccae2b53461e9f2936cf789c2b5d, State: Running, Role: LEADER
I20260620 14:55:04.065454 25868 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3ccae2b53461e9f2936cf789c2b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } }
I20260620 14:55:04.065546 25863 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:04.065788 25868 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4f3ccae2b53461e9f2936cf789c2b5d. Latest consensus state: current_term: 1 leader_uuid: "f4f3ccae2b53461e9f2936cf789c2b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3ccae2b53461e9f2936cf789c2b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } } }
I20260620 14:55:04.065879 25868 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:04.065804 25869 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4f3ccae2b53461e9f2936cf789c2b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f3ccae2b53461e9f2936cf789c2b5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39293 } } }
I20260620 14:55:04.066100 25872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:04.066715 25872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:04.066105 25869 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:04.069113 25872 catalog_manager.cc:1347] Generated new cluster ID: d04dd44b8a26495995fc586d6719d456
I20260620 14:55:04.069191 25872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:04.069418 25866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:04.057083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56593 (local address 127.0.0.1:39293)
0620 14:55:04.057282 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:04.057294 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:04.057333 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:55:04.057926 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:04.057936 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:04.058005 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:04.058193 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:04.058777 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:04.060486 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:04.061476 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:04.062228 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:04.062372 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:04.065093 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:04.065098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:04.065099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:04.065114 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:04.067315 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:04.067834 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:04.067837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:04.067838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:04.067946 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:04.068290 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:04.068291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:04.068291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:04.068617 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:04.068791 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:04.068959 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:55:04.069197 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260620 14:55:04.070453 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:04.080089 25866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:04.071999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51467 (local address 127.0.0.1:39293)
0620 14:55:04.072041 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:04.072043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:04.072200 (+ 157us) server_negotiation.cc:407] Connection header received
0620 14:55:04.072282 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:04.072283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:04.072294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:04.072366 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:04.073128 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:04.073948 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:04.075023 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:04.075389 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:04.075461 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:04.077028 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:04.077030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:04.077030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:04.077037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:04.078798 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:04.079231 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:04.079232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:04.079233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:04.079292 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:04.079523 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:04.079523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:04.079524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:04.079660 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:04.079761 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:04.079920 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 14:55:04.079985 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:55:04.091468 25872 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:04.092296 25872 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:04.106004 25872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4f3ccae2b53461e9f2936cf789c2b5d: Generated new TSK 0
I20260620 14:55:04.106316 25872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:04.115518 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 25795
W20260620 14:55:04.120131 25780 connection.cc:535] client connection to 127.0.0.1:39293 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39293): Connection reset by peer (error 104)
2026-06-20T14:55:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:04 dist-test-slave-2m5f krb5kdc[25894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:04 dist-test-slave-2m5f krb5kdc[25894](info): set up 2 sockets
Jun 20 14:55:04 dist-test-slave-2m5f krb5kdc[25894](info): commencing operation
krb5kdc: starting...
W20260620 14:55:05.160403 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:05 dist-test-slave-2m5f krb5kdc[25894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:05.287719 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:05.376104 25907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:05.376364 25907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:05.376389 25907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:05.377581 25907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:05.377622 25907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:05.377637 25907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:05.377650 25907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:05.377661 25907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:05.379108 25907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:05.379308 25907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:05.379482 25907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:05.384582 25912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:05.384841 25915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:05.384687 25913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:05.384953 25907 server_base.cc:1034] running on GCE node
I20260620 14:55:05.385299 25907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:05.385627 25907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:05.386794 25907 hybrid_clock.cc:648] HybridClock initialized: now 1781967305386767 us; error 38 us; skew 500 ppm
Jun 20 14:55:05 dist-test-slave-2m5f krb5kdc[25894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:05.389582 25907 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:05.390034 25907 webserver.cc:460] Webserver started at http://127.0.0.1:35101/ using document root <none> and password file <none>
I20260620 14:55:05.390277 25907 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:05.390347 25907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:05.390496 25907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:05.391599 25907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "4c39efae129146c28cc216e695e2869d"
format_stamp: "Formatted at 2026-06-20 14:55:05 on dist-test-slave-2m5f"
server_key: "734e37457993af3d6dfdc906c8258e4f"
server_key_iv: "ece24dccb2188cb47bfa8cbe2e9a6c23"
server_key_version: "encryptionkey@0"
I20260620 14:55:05.392006 25907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "4c39efae129146c28cc216e695e2869d"
format_stamp: "Formatted at 2026-06-20 14:55:05 on dist-test-slave-2m5f"
server_key: "734e37457993af3d6dfdc906c8258e4f"
server_key_iv: "ece24dccb2188cb47bfa8cbe2e9a6c23"
server_key_version: "encryptionkey@0"
I20260620 14:55:05.393776 25907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:55:05.394608 25922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:05.394819 25907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:55:05.394924 25907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "4c39efae129146c28cc216e695e2869d"
format_stamp: "Formatted at 2026-06-20 14:55:05 on dist-test-slave-2m5f"
server_key: "734e37457993af3d6dfdc906c8258e4f"
server_key_iv: "ece24dccb2188cb47bfa8cbe2e9a6c23"
server_key_version: "encryptionkey@0"
I20260620 14:55:05.395015 25907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:05.424147 25907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:05.427006 25907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:05.427214 25907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:05.431300 25907 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35765
I20260620 14:55:05.431351 25974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35765 every 8 connection(s)
I20260620 14:55:05.431730 25907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:05.432857 25975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:05.433394 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 25907
I20260620 14:55:05.433533 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:05.433749 5594 external_mini_cluster.cc:1428] Setting key 59641d6f53b9851747d7e32ce20fa465
I20260620 14:55:05.435823 25975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d: Bootstrap starting.
I20260620 14:55:05.436502 25975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:05.436843 25975 log.cc:824] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:05.437629 25975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d: No bootstrap required, opened a new log
I20260620 14:55:05.439502 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:05.434323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51897 (local address 127.0.0.1:35765)
0620 14:55:05.434500 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:05.434510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:05.434537 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:55:05.434952 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:05.434959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:05.435085 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:05.435218 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:05.435466 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:05.436479 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:05.437535 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:05.437993 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:05.438081 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:05.438854 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:05.438857 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:05.438858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:05.439023 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:05.439053 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:05.439064 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:05.439184 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:05.439247 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:55:05.439295 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260620 14:55:05.439985 25975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c39efae129146c28cc216e695e2869d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } }
I20260620 14:55:05.440155 25975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:05.440222 25975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c39efae129146c28cc216e695e2869d, State: Initialized, Role: FOLLOWER
I20260620 14:55:05.440379 25975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c39efae129146c28cc216e695e2869d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } }
I20260620 14:55:05.440474 25975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:05.440464 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:05.440521 25975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:05.440567 25975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:05.441121 25975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c39efae129146c28cc216e695e2869d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } }
I20260620 14:55:05.441206 25975 leader_election.cc:302] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c39efae129146c28cc216e695e2869d; no voters:
I20260620 14:55:05.441403 25975 leader_election.cc:288] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:05.441440 25981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:05.441612 25981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [term 1 LEADER]: Becoming Leader. State: Replica: 4c39efae129146c28cc216e695e2869d, State: Running, Role: LEADER
I20260620 14:55:05.441673 25975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:05.441744 25981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c39efae129146c28cc216e695e2869d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } }
I20260620 14:55:05.442129 25984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c39efae129146c28cc216e695e2869d. Latest consensus state: current_term: 1 leader_uuid: "4c39efae129146c28cc216e695e2869d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c39efae129146c28cc216e695e2869d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } } }
I20260620 14:55:05.442243 25984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:05.442127 25986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c39efae129146c28cc216e695e2869d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c39efae129146c28cc216e695e2869d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } } }
I20260620 14:55:05.442488 25986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:05.442873 25992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:05.443418 25992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:05.444839 25992 catalog_manager.cc:1347] Generated new cluster ID: 81a266855e2d4928a68dabeb08e75512
I20260620 14:55:05.444896 25992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:05.445803 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:05.441851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53353 (local address 127.0.0.1:35765)
0620 14:55:05.441892 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:05.441894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:05.442239 (+ 345us) server_negotiation.cc:407] Connection header received
0620 14:55:05.442351 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:05.442352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:05.442364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:05.442398 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:05.442620 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:05.443405 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:05.444464 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:05.444705 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:05.444790 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:05.445566 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:05.445568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:05.445568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:05.445573 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:05.445587 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:05.445591 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:05.445659 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:05.445719 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:55:05.445729 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:55:05.490635 25992 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:05.491307 25992 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:05.495836 25992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c39efae129146c28cc216e695e2869d: Generated new TSK 0
I20260620 14:55:05.496058 25992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:05.506431 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 25907
W20260620 14:55:05.511128 25892 connection.cc:535] client connection to 127.0.0.1:35765 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35765): Connection reset by peer (error 104)
2026-06-20T14:55:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:05 dist-test-slave-2m5f krb5kdc[26006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:05 dist-test-slave-2m5f krb5kdc[26006](info): set up 2 sockets
Jun 20 14:55:05 dist-test-slave-2m5f krb5kdc[26006](info): commencing operation
krb5kdc: starting...
W20260620 14:55:06.548386 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:06 dist-test-slave-2m5f krb5kdc[26006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:06.674830 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:06.758138 26019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:06.758373 26019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:06.758399 26019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:06.759644 26019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:06.759686 26019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:06.759701 26019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:06.759713 26019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:06.759725 26019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:06.761263 26019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34191
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:06.761469 26019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:06.761655 26019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:06.766546 26024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:06.766541 26025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:06.766631 26019 server_base.cc:1034] running on GCE node
W20260620 14:55:06.766541 26027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:06.766999 26019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:06.767295 26019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:06.768471 26019 hybrid_clock.cc:648] HybridClock initialized: now 1781967306768443 us; error 44 us; skew 500 ppm
Jun 20 14:55:06 dist-test-slave-2m5f krb5kdc[26006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:06.776360 26019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:06.776870 26019 webserver.cc:460] Webserver started at http://127.0.0.1:42545/ using document root <none> and password file <none>
I20260620 14:55:06.777124 26019 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:06.777172 26019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:06.777323 26019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:06.778498 26019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "9e6d35c8a2f24b0ea115157ec022d197"
format_stamp: "Formatted at 2026-06-20 14:55:06 on dist-test-slave-2m5f"
server_key: "d37d1b7567b626524a62d2a4c6a2d320"
server_key_iv: "280d7d810145c98fed115298ef8f0d55"
server_key_version: "encryptionkey@0"
I20260620 14:55:06.778882 26019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "9e6d35c8a2f24b0ea115157ec022d197"
format_stamp: "Formatted at 2026-06-20 14:55:06 on dist-test-slave-2m5f"
server_key: "d37d1b7567b626524a62d2a4c6a2d320"
server_key_iv: "280d7d810145c98fed115298ef8f0d55"
server_key_version: "encryptionkey@0"
I20260620 14:55:06.780326 26019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:55:06.781153 26034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:06.781338 26019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:06.781445 26019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "9e6d35c8a2f24b0ea115157ec022d197"
format_stamp: "Formatted at 2026-06-20 14:55:06 on dist-test-slave-2m5f"
server_key: "d37d1b7567b626524a62d2a4c6a2d320"
server_key_iv: "280d7d810145c98fed115298ef8f0d55"
server_key_version: "encryptionkey@0"
I20260620 14:55:06.781523 26019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:06.785053 26019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:06.787248 26019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:06.787412 26019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:06.791700 26019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34191
I20260620 14:55:06.791736 26086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34191 every 8 connection(s)
I20260620 14:55:06.792109 26019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:06.793167 26087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:06.795465 26087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197: Bootstrap starting.
I20260620 14:55:06.796012 26087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:06.796355 26087 log.cc:824] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:06.797062 26087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197: No bootstrap required, opened a new log
I20260620 14:55:06.798660 26087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6d35c8a2f24b0ea115157ec022d197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34191 } }
I20260620 14:55:06.798861 26087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:06.798887 26087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e6d35c8a2f24b0ea115157ec022d197, State: Initialized, Role: FOLLOWER
I20260620 14:55:06.799047 26087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6d35c8a2f24b0ea115157ec022d197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34191 } }
I20260620 14:55:06.799129 26087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:06.799152 26087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:06.799182 26087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:06.799713 26087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6d35c8a2f24b0ea115157ec022d197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34191 } }
I20260620 14:55:06.799808 26087 leader_election.cc:302] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e6d35c8a2f24b0ea115157ec022d197; no voters:
I20260620 14:55:06.799952 26087 leader_election.cc:288] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:06.800027 26090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:06.800230 26090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [term 1 LEADER]: Becoming Leader. State: Replica: 9e6d35c8a2f24b0ea115157ec022d197, State: Running, Role: LEADER
I20260620 14:55:06.800267 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 26019
I20260620 14:55:06.800302 26087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:06.800395 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:06.800379 26090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6d35c8a2f24b0ea115157ec022d197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34191 } }
I20260620 14:55:06.800602 5594 external_mini_cluster.cc:1428] Setting key f957315f4d9c0c786048f88eec88f90a
I20260620 14:55:06.800777 26091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e6d35c8a2f24b0ea115157ec022d197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6d35c8a2f24b0ea115157ec022d197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34191 } } }
I20260620 14:55:06.800756 26090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e6d35c8a2f24b0ea115157ec022d197. Latest consensus state: current_term: 1 leader_uuid: "9e6d35c8a2f24b0ea115157ec022d197" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6d35c8a2f24b0ea115157ec022d197" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34191 } } }
I20260620 14:55:06.800837 26091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:06.800845 26090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:06.801281 26094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:06.801873 26094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:06.803422 26097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:06.801067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38729 (local address 127.0.0.1:34191)
0620 14:55:06.801623 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:06.801634 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:06.801666 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:55:06.801793 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:06.801801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:06.801834 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:06.802293 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:06.802748 (+ 455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:06.802752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:06.802752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:06.802756 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:06.802811 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:06.802823 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:06.802909 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:06.802986 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 14:55:06.803029 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":70,"threads_started":1}
I20260620 14:55:06.803746 26094 catalog_manager.cc:1347] Generated new cluster ID: 82f209ef3e1a4c0f9c45675dc08cefa2
I20260620 14:55:06.803826 26094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:06.804478 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:06.807147 26097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:06.805971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40063 (local address 127.0.0.1:34191)
0620 14:55:06.806002 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:06.806003 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:06.806179 (+ 176us) server_negotiation.cc:407] Connection header received
0620 14:55:06.806213 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:06.806213 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:06.806219 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:06.806288 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:06.806922 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:06.806923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:06.806923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:06.806927 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:06.806944 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:06.806948 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:06.807011 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:06.807060 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 14:55:06.807074 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:55:06.812098 26094 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:06.812295 26094 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:06.817282 26094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e6d35c8a2f24b0ea115157ec022d197: Generated new TSK 0
I20260620 14:55:06.817481 26094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:06.822597 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 26019
2026-06-20T14:55:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:06 dist-test-slave-2m5f krb5kdc[26118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:06 dist-test-slave-2m5f krb5kdc[26118](info): set up 2 sockets
Jun 20 14:55:06 dist-test-slave-2m5f krb5kdc[26118](info): commencing operation
krb5kdc: starting...
W20260620 14:55:07.868435 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:07 dist-test-slave-2m5f krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:07.993041 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:08.075851 26131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:08.076107 26131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:08.076146 26131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:08.077332 26131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:08.077407 26131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:08.077435 26131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:08.077466 26131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:08.077490 26131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:08.079027 26131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:08.079309 26131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:08.079540 26131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:08.084478 26137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:08.084470 26136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:08.084470 26139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:08.084731 26131 server_base.cc:1034] running on GCE node
I20260620 14:55:08.085021 26131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:08.085302 26131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:08.086468 26131 hybrid_clock.cc:648] HybridClock initialized: now 1781967308086462 us; error 36 us; skew 500 ppm
Jun 20 14:55:08 dist-test-slave-2m5f krb5kdc[26118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:08.089006 26131 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:08.089457 26131 webserver.cc:460] Webserver started at http://127.0.0.1:40607/ using document root <none> and password file <none>
I20260620 14:55:08.089656 26131 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:08.089697 26131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:08.089785 26131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:08.090804 26131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "285027f28be64935bd3cc27d8355d366"
format_stamp: "Formatted at 2026-06-20 14:55:08 on dist-test-slave-2m5f"
server_key: "8c5a58b07643968a0891b940695e38c2"
server_key_iv: "b37ab517e328edf44de8310f499915e1"
server_key_version: "encryptionkey@0"
I20260620 14:55:08.091266 26131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "285027f28be64935bd3cc27d8355d366"
format_stamp: "Formatted at 2026-06-20 14:55:08 on dist-test-slave-2m5f"
server_key: "8c5a58b07643968a0891b940695e38c2"
server_key_iv: "b37ab517e328edf44de8310f499915e1"
server_key_version: "encryptionkey@0"
I20260620 14:55:08.097908 26131 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.003s
I20260620 14:55:08.100274 26146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:08.100495 26131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 14:55:08.100601 26131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "285027f28be64935bd3cc27d8355d366"
format_stamp: "Formatted at 2026-06-20 14:55:08 on dist-test-slave-2m5f"
server_key: "8c5a58b07643968a0891b940695e38c2"
server_key_iv: "b37ab517e328edf44de8310f499915e1"
server_key_version: "encryptionkey@0"
I20260620 14:55:08.100689 26131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:08.135533 26131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:08.138415 26131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:08.138540 26131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:08.142495 26131 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40711
I20260620 14:55:08.142570 26198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40711 every 8 connection(s)
I20260620 14:55:08.142946 26131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:08.143900 26199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:08.146561 26199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366: Bootstrap starting.
I20260620 14:55:08.147169 26199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:08.147468 26199 log.cc:824] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:08.148353 26199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366: No bootstrap required, opened a new log
I20260620 14:55:08.148393 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 26131
I20260620 14:55:08.148514 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:08.148708 5594 external_mini_cluster.cc:1428] Setting key a670729a5c69bca022bb936a437412e8
I20260620 14:55:08.150355 26199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285027f28be64935bd3cc27d8355d366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } }
I20260620 14:55:08.150527 26199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:08.150595 26199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285027f28be64935bd3cc27d8355d366, State: Initialized, Role: FOLLOWER
I20260620 14:55:08.150758 26199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285027f28be64935bd3cc27d8355d366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } }
I20260620 14:55:08.150902 26199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:08.150947 26199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:08.151001 26199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:08.151902 26199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285027f28be64935bd3cc27d8355d366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } }
I20260620 14:55:08.152020 26199 leader_election.cc:302] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 285027f28be64935bd3cc27d8355d366; no voters:
I20260620 14:55:08.152257 26199 leader_election.cc:288] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:08.152303 26204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:08.152486 26204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [term 1 LEADER]: Becoming Leader. State: Replica: 285027f28be64935bd3cc27d8355d366, State: Running, Role: LEADER
I20260620 14:55:08.152534 26199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:08.152629 26204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285027f28be64935bd3cc27d8355d366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } }
I20260620 14:55:08.153759 26206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 285027f28be64935bd3cc27d8355d366. Latest consensus state: current_term: 1 leader_uuid: "285027f28be64935bd3cc27d8355d366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285027f28be64935bd3cc27d8355d366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } } }
I20260620 14:55:08.153879 26206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [sys.catalog]: This master's current role is: LEADER
W20260620 14:55:08.154328 26218 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:55:08.154376 26218 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:55:08.154444 26220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:08.154596 26220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:08.154976 26205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "285027f28be64935bd3cc27d8355d366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285027f28be64935bd3cc27d8355d366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40711 } } }
I20260620 14:55:08.155081 26205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366 [sys.catalog]: This master's current role is: LEADER
Jun 20 14:55:08 dist-test-slave-2m5f krb5kdc[26118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:55:08.157683 26220 catalog_manager.cc:1347] Generated new cluster ID: 9ab1a5df80534a02804f3150c53e8bf0
I20260620 14:55:08.157738 26220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:08.161537 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:08.149235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41247 (local address 127.0.0.1:40711)
0620 14:55:08.149481 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:08.149491 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:08.149528 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:55:08.150402 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:08.150410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:08.150464 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:08.150726 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:08.151464 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:08.152771 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:08.153978 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:08.154617 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:08.154759 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:08.157798 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:08.157802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:08.157802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:08.157830 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:08.159540 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:08.159967 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:08.159970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:08.159971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:08.160085 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:08.160333 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:08.160334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:08.160334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:08.160583 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:08.161193 (+ 610us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:08.161263 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:55:08.161376 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":159,"threads_started":1}
I20260620 14:55:08.162623 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:08.180437 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:08.164052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54413 (local address 127.0.0.1:40711)
0620 14:55:08.164094 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:08.164096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:08.168459 (+ 4363us) server_negotiation.cc:407] Connection header received
0620 14:55:08.168545 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:08.168546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:08.168558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:08.168609 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:08.173359 (+ 4750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:08.174145 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:08.175205 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:08.175549 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:08.175588 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:08.177105 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:08.177106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:08.177107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:08.177114 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:08.179239 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:08.179607 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:08.179608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:08.179608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:08.179654 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:08.179817 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:08.179817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:08.179818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:08.179936 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:08.180026 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:08.180231 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:55:08.180334 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 14:55:08.202620 26220 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:08.203459 26220 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:08.207907 26220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 285027f28be64935bd3cc27d8355d366: Generated new TSK 0
I20260620 14:55:08.208135 26220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:08.210960 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 26131
2026-06-20T14:55:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:08 dist-test-slave-2m5f krb5kdc[26230](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:08 dist-test-slave-2m5f krb5kdc[26230](info): set up 2 sockets
Jun 20 14:55:08 dist-test-slave-2m5f krb5kdc[26230](info): commencing operation
krb5kdc: starting...
W20260620 14:55:09.264514 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:09 dist-test-slave-2m5f krb5kdc[26230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:09.390918 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:09.477685 26243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:09.477937 26243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:09.477965 26243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:09.479899 26243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:09.479965 26243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:09.480026 26243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:09.480057 26243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:09.480088 26243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:09.481839 26243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:09.482127 26243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:09.482427 26243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:09.487959 26249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:09.487922 26248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:09.488142 26243 server_base.cc:1034] running on GCE node
W20260620 14:55:09.487929 26251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:09.488494 26243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:09.488780 26243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:09.489943 26243 hybrid_clock.cc:648] HybridClock initialized: now 1781967309489912 us; error 43 us; skew 500 ppm
Jun 20 14:55:09 dist-test-slave-2m5f krb5kdc[26230](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:09.492769 26243 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:09.493286 26243 webserver.cc:460] Webserver started at http://127.0.0.1:45367/ using document root <none> and password file <none>
I20260620 14:55:09.493541 26243 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:09.493593 26243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:09.493732 26243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:09.495028 26243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "99f028c4d5924dc086e564e6460cdb9f"
format_stamp: "Formatted at 2026-06-20 14:55:09 on dist-test-slave-2m5f"
server_key: "117568073761e5aba19d69196313fe5d"
server_key_iv: "c5e5a21d7cc975ebaa0abb5c041f467f"
server_key_version: "encryptionkey@0"
I20260620 14:55:09.495420 26243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "99f028c4d5924dc086e564e6460cdb9f"
format_stamp: "Formatted at 2026-06-20 14:55:09 on dist-test-slave-2m5f"
server_key: "117568073761e5aba19d69196313fe5d"
server_key_iv: "c5e5a21d7cc975ebaa0abb5c041f467f"
server_key_version: "encryptionkey@0"
I20260620 14:55:09.497000 26243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 14:55:09.497893 26258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:09.498145 26243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:55:09.498257 26243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "99f028c4d5924dc086e564e6460cdb9f"
format_stamp: "Formatted at 2026-06-20 14:55:09 on dist-test-slave-2m5f"
server_key: "117568073761e5aba19d69196313fe5d"
server_key_iv: "c5e5a21d7cc975ebaa0abb5c041f467f"
server_key_version: "encryptionkey@0"
I20260620 14:55:09.498337 26243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:09.518733 26243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:09.521518 26243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:09.521703 26243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:09.526117 26243 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38111
I20260620 14:55:09.526206 26310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38111 every 8 connection(s)
I20260620 14:55:09.526582 26243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:09.527621 26311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:09.530145 26311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f: Bootstrap starting.
I20260620 14:55:09.530776 26311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:09.531075 26311 log.cc:824] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:09.531876 26311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f: No bootstrap required, opened a new log
I20260620 14:55:09.533659 26311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f028c4d5924dc086e564e6460cdb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260620 14:55:09.533823 26311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:09.533847 26311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f028c4d5924dc086e564e6460cdb9f, State: Initialized, Role: FOLLOWER
I20260620 14:55:09.534014 26311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f028c4d5924dc086e564e6460cdb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260620 14:55:09.534086 26311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:09.534109 26311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:09.534181 26311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:09.534751 26311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f028c4d5924dc086e564e6460cdb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260620 14:55:09.534848 26311 leader_election.cc:302] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99f028c4d5924dc086e564e6460cdb9f; no voters:
I20260620 14:55:09.535070 26311 leader_election.cc:288] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:09.535163 26314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:09.535339 26311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:09.535437 26314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [term 1 LEADER]: Becoming Leader. State: Replica: 99f028c4d5924dc086e564e6460cdb9f, State: Running, Role: LEADER
I20260620 14:55:09.535579 26314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f028c4d5924dc086e564e6460cdb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260620 14:55:09.536293 26316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99f028c4d5924dc086e564e6460cdb9f. Latest consensus state: current_term: 1 leader_uuid: "99f028c4d5924dc086e564e6460cdb9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f028c4d5924dc086e564e6460cdb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } } }
I20260620 14:55:09.536376 26316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:09.536494 26315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99f028c4d5924dc086e564e6460cdb9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99f028c4d5924dc086e564e6460cdb9f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } } }
I20260620 14:55:09.536569 26315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:09.536659 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 26243
I20260620 14:55:09.536790 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:09.536942 26324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:09.536990 5594 external_mini_cluster.cc:1428] Setting key 3b5f422d1d4bcf818bb743334939d477
I20260620 14:55:09.537482 26324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:09.539049 26324 catalog_manager.cc:1347] Generated new cluster ID: d6d3fd87c04247e09077447063779dff
I20260620 14:55:09.539106 26324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:09.542692 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:09.537663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55655 (local address 127.0.0.1:38111)
0620 14:55:09.537840 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:09.537854 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:09.538101 (+ 247us) server_negotiation.cc:407] Connection header received
0620 14:55:09.538305 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:09.538314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:09.538372 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:09.538515 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:09.538718 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:09.539755 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:09.540681 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:09.541156 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:09.541291 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:09.542018 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:09.542021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:09.542021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:09.542203 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:09.542234 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:09.542246 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:09.542322 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:09.542397 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:55:09.542466 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260620 14:55:09.543809 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:09.552009 26324 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:09.552636 26324 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:09.555380 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:09.545255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60397 (local address 127.0.0.1:38111)
0620 14:55:09.545300 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:09.545302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:09.551649 (+ 6347us) server_negotiation.cc:407] Connection header received
0620 14:55:09.551737 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:09.551738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:09.551752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:09.551807 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:09.552053 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:09.552908 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:09.553960 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:09.554282 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:09.554362 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:09.555099 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:09.555101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:09.555101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:09.555108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:09.555125 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:09.555130 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:09.555190 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:09.555251 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 14:55:09.555296 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":23}
I20260620 14:55:09.561966 26324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99f028c4d5924dc086e564e6460cdb9f: Generated new TSK 0
I20260620 14:55:09.562186 26324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:09.570472 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 26243
2026-06-20T14:55:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:09 dist-test-slave-2m5f krb5kdc[26342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:09 dist-test-slave-2m5f krb5kdc[26342](info): set up 2 sockets
Jun 20 14:55:09 dist-test-slave-2m5f krb5kdc[26342](info): commencing operation
krb5kdc: starting...
W20260620 14:55:10.620515 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:10 dist-test-slave-2m5f krb5kdc[26342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:10.743314 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:10.830631 26355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:10.830853 26355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:10.830878 26355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:10.832070 26355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:10.832122 26355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:10.832136 26355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:10.832149 26355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:10.832217 26355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:10.833793 26355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:10.834061 26355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:10.834311 26355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:10.839468 26363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:10.839517 26355 server_base.cc:1034] running on GCE node
W20260620 14:55:10.839600 26360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:10.839486 26361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:10.840057 26355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:10.840404 26355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:10.841607 26355 hybrid_clock.cc:648] HybridClock initialized: now 1781967310841598 us; error 52 us; skew 500 ppm
Jun 20 14:55:10 dist-test-slave-2m5f krb5kdc[26342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:10.844416 26355 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:10.845073 26355 webserver.cc:460] Webserver started at http://127.0.0.1:42867/ using document root <none> and password file <none>
I20260620 14:55:10.845412 26355 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:10.845497 26355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:10.845641 26355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:10.847092 26355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "7ef93b71cfb446029d5071ec19f88efa"
format_stamp: "Formatted at 2026-06-20 14:55:10 on dist-test-slave-2m5f"
server_key: "8da232180a9277704a7f749db65e0e85"
server_key_iv: "b628525baacfd4e9d8c70be8d1bf65e1"
server_key_version: "encryptionkey@0"
I20260620 14:55:10.847551 26355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "7ef93b71cfb446029d5071ec19f88efa"
format_stamp: "Formatted at 2026-06-20 14:55:10 on dist-test-slave-2m5f"
server_key: "8da232180a9277704a7f749db65e0e85"
server_key_iv: "b628525baacfd4e9d8c70be8d1bf65e1"
server_key_version: "encryptionkey@0"
I20260620 14:55:10.849503 26355 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 14:55:10.850490 26370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:10.850675 26355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:10.850816 26355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "7ef93b71cfb446029d5071ec19f88efa"
format_stamp: "Formatted at 2026-06-20 14:55:10 on dist-test-slave-2m5f"
server_key: "8da232180a9277704a7f749db65e0e85"
server_key_iv: "b628525baacfd4e9d8c70be8d1bf65e1"
server_key_version: "encryptionkey@0"
I20260620 14:55:10.850906 26355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:10.855697 26355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:10.858413 26355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:10.858592 26355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:10.863773 26355 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33317
I20260620 14:55:10.863843 26422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33317 every 8 connection(s)
I20260620 14:55:10.864312 26355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:10.865624 26423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:10.868798 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 26355
I20260620 14:55:10.868942 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:10.869135 5594 external_mini_cluster.cc:1428] Setting key a788183220b85d5a60555eb79c7424af
I20260620 14:55:10.869602 26423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa: Bootstrap starting.
I20260620 14:55:10.870370 26423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:10.870743 26423 log.cc:824] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:10.871601 26423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa: No bootstrap required, opened a new log
I20260620 14:55:10.872130 26426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:10.869696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42395 (local address 127.0.0.1:33317)
0620 14:55:10.869861 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:10.869870 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:10.869903 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:55:10.870397 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:10.870407 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:10.870499 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:10.870673 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:10.871456 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:10.871460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:10.871460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:10.871465 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:10.871509 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:10.871523 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:10.871601 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:10.871880 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:55:10.871924 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260620 14:55:10.873145 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:10.874738 26423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef93b71cfb446029d5071ec19f88efa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } }
I20260620 14:55:10.874969 26423 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:10.875002 26423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef93b71cfb446029d5071ec19f88efa, State: Initialized, Role: FOLLOWER
I20260620 14:55:10.875152 26423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef93b71cfb446029d5071ec19f88efa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } }
I20260620 14:55:10.875265 26423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:10.875298 26423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:10.875348 26423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:10.876119 26423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef93b71cfb446029d5071ec19f88efa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } }
I20260620 14:55:10.876278 26423 leader_election.cc:302] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ef93b71cfb446029d5071ec19f88efa; no voters:
I20260620 14:55:10.876447 26423 leader_election.cc:288] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:10.876495 26433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:10.876675 26433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [term 1 LEADER]: Becoming Leader. State: Replica: 7ef93b71cfb446029d5071ec19f88efa, State: Running, Role: LEADER
I20260620 14:55:10.876734 26423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:10.876825 26433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef93b71cfb446029d5071ec19f88efa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } }
I20260620 14:55:10.877091 26426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:10.875805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44817 (local address 127.0.0.1:33317)
0620 14:55:10.875862 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:10.875864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:10.875872 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:55:10.875892 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:10.875893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:10.875899 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:10.875952 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:10.876660 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:10.876662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:10.876662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:10.876666 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:10.876680 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:10.876684 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:10.877001 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:10.877020 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 14:55:10.877033 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 14:55:10.877272 26433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ef93b71cfb446029d5071ec19f88efa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef93b71cfb446029d5071ec19f88efa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } } }
I20260620 14:55:10.877391 26433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:10.877540 26433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ef93b71cfb446029d5071ec19f88efa. Latest consensus state: current_term: 1 leader_uuid: "7ef93b71cfb446029d5071ec19f88efa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ef93b71cfb446029d5071ec19f88efa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33317 } } }
I20260620 14:55:10.877622 26433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa [sys.catalog]: This master's current role is: LEADER
W20260620 14:55:10.878558 26448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:55:10.880595 26448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:55:10.880713 26442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:10.880978 26442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:10.883092 26442 catalog_manager.cc:1347] Generated new cluster ID: f257861d0386403aa92421b441d19221
I20260620 14:55:10.883149 26442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:10.903491 26442 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:10.903694 26442 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:10.907794 26442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ef93b71cfb446029d5071ec19f88efa: Generated new TSK 0
I20260620 14:55:10.908017 26442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:10.911669 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 26355
2026-06-20T14:55:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:10 dist-test-slave-2m5f krb5kdc[26454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:10 dist-test-slave-2m5f krb5kdc[26454](info): set up 2 sockets
Jun 20 14:55:10 dist-test-slave-2m5f krb5kdc[26454](info): commencing operation
krb5kdc: starting...
W20260620 14:55:11.972548 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:12 dist-test-slave-2m5f krb5kdc[26454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:12.099156 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:12.185936 26467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:12.186175 26467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:12.186201 26467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:12.187399 26467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:12.187443 26467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:12.187458 26467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:12.187470 26467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:12.187482 26467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:12.188994 26467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:12.189216 26467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:12.189402 26467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:12.194520 26473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:12.194514 26475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:12.194626 26467 server_base.cc:1034] running on GCE node
W20260620 14:55:12.194525 26472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:12.195080 26467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:12.195396 26467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:12.196568 26467 hybrid_clock.cc:648] HybridClock initialized: now 1781967312196551 us; error 51 us; skew 500 ppm
Jun 20 14:55:12 dist-test-slave-2m5f krb5kdc[26454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:12.204667 26467 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:12.205168 26467 webserver.cc:460] Webserver started at http://127.0.0.1:41333/ using document root <none> and password file <none>
I20260620 14:55:12.205467 26467 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:12.205543 26467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:12.205704 26467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:12.206833 26467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "f62ce352660a47748386cf2e2af09879"
format_stamp: "Formatted at 2026-06-20 14:55:12 on dist-test-slave-2m5f"
server_key: "a4239ddcd0bf440487ba662530286fa9"
server_key_iv: "efbcfc66c405f1b8a5c5e4dec5ba6988"
server_key_version: "encryptionkey@0"
I20260620 14:55:12.207204 26467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "f62ce352660a47748386cf2e2af09879"
format_stamp: "Formatted at 2026-06-20 14:55:12 on dist-test-slave-2m5f"
server_key: "a4239ddcd0bf440487ba662530286fa9"
server_key_iv: "efbcfc66c405f1b8a5c5e4dec5ba6988"
server_key_version: "encryptionkey@0"
I20260620 14:55:12.208642 26467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:55:12.209517 26482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:12.209712 26467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:12.209821 26467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "f62ce352660a47748386cf2e2af09879"
format_stamp: "Formatted at 2026-06-20 14:55:12 on dist-test-slave-2m5f"
server_key: "a4239ddcd0bf440487ba662530286fa9"
server_key_iv: "efbcfc66c405f1b8a5c5e4dec5ba6988"
server_key_version: "encryptionkey@0"
I20260620 14:55:12.209900 26467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:12.227253 26467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:12.229424 26467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:12.229593 26467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:12.234146 26467 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36805
I20260620 14:55:12.234184 26534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36805 every 8 connection(s)
I20260620 14:55:12.234570 26467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:12.234856 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 26467
I20260620 14:55:12.234978 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:12.235198 5594 external_mini_cluster.cc:1428] Setting key 8e09b7f6fa956e2ead904c0f1a024583
I20260620 14:55:12.235853 26535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:12.238442 26535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879: Bootstrap starting.
I20260620 14:55:12.239061 26535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:12.239362 26535 log.cc:824] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:12.240217 26535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879: No bootstrap required, opened a new log
Jun 20 14:55:12 dist-test-slave-2m5f krb5kdc[26454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:55:12.242709 26535 raft_consensus.cc:348] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62ce352660a47748386cf2e2af09879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260620 14:55:12.242892 26535 raft_consensus.cc:374] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:12.242928 26535 raft_consensus.cc:729] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62ce352660a47748386cf2e2af09879, State: Initialized, Role: FOLLOWER
I20260620 14:55:12.243074 26535 consensus_queue.cc:260] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62ce352660a47748386cf2e2af09879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260620 14:55:12.243165 26535 raft_consensus.cc:388] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:12.243206 26535 raft_consensus.cc:482] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:12.243265 26535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:12.244024 26535 raft_consensus.cc:504] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62ce352660a47748386cf2e2af09879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260620 14:55:12.244148 26535 leader_election.cc:302] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f62ce352660a47748386cf2e2af09879; no voters:
I20260620 14:55:12.244361 26535 leader_election.cc:288] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:12.244438 26540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:12.244599 26540 raft_consensus.cc:686] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [term 1 LEADER]: Becoming Leader. State: Replica: f62ce352660a47748386cf2e2af09879, State: Running, Role: LEADER
I20260620 14:55:12.244637 26535 sys_catalog.cc:564] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:12.244712 26540 consensus_queue.cc:237] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62ce352660a47748386cf2e2af09879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260620 14:55:12.245018 26541 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f62ce352660a47748386cf2e2af09879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62ce352660a47748386cf2e2af09879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } } }
I20260620 14:55:12.245163 26541 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:12.245397 26542 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f62ce352660a47748386cf2e2af09879. Latest consensus state: current_term: 1 leader_uuid: "f62ce352660a47748386cf2e2af09879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62ce352660a47748386cf2e2af09879" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } } }
I20260620 14:55:12.245476 26542 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:12.245656 26547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:12.246163 26547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:12.247169 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:12.235707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:50059 (local address 127.0.0.1:36805)
0620 14:55:12.235889 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:12.235900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:12.235940 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:55:12.236458 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:12.236465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:12.236519 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:12.236678 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:12.237303 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:12.238209 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:12.239091 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:12.239621 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:12.239728 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:12.242539 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:12.242553 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:12.242553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:12.242567 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:12.244840 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:12.245409 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:12.245414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:12.245414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:12.245533 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:12.245806 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:12.245807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:12.245808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:12.246206 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:12.246859 (+ 653us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:12.246929 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:55:12.247007 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260620 14:55:12.248288 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:12.248343 26547 catalog_manager.cc:1347] Generated new cluster ID: 05f40d7ac1d744a495a84e607a96d369
I20260620 14:55:12.248396 26547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:12.257223 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:12.249631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:35319 (local address 127.0.0.1:36805)
0620 14:55:12.249664 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:12.249666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:12.249782 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:55:12.249838 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:12.249839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:12.249849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:12.249903 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:12.250524 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:12.251258 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:12.252384 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:12.252715 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:12.252784 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:12.254342 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:12.254343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:12.254344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:12.254349 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:12.255908 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:12.256376 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:12.256377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:12.256377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:12.256434 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:12.256676 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:12.256677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:12.256677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:12.256797 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:12.256888 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:12.257064 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 14:55:12.257129 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:55:12.260236 26547 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:12.260998 26547 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:12.265832 26547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f62ce352660a47748386cf2e2af09879: Generated new TSK 0
I20260620 14:55:12.266088 26547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:12.274564 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 26467
W20260620 14:55:12.278903 26452 connection.cc:535] client connection to 127.0.0.1:36805 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36805): Connection reset by peer (error 104)
2026-06-20T14:55:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:12 dist-test-slave-2m5f krb5kdc[26566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:12 dist-test-slave-2m5f krb5kdc[26566](info): set up 2 sockets
Jun 20 14:55:12 dist-test-slave-2m5f krb5kdc[26566](info): commencing operation
krb5kdc: starting...
W20260620 14:55:13.380435 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:13 dist-test-slave-2m5f krb5kdc[26566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:13.507535 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:13.589918 26579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:13.590193 26579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:13.590235 26579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:13.591482 26579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:13.591559 26579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:13.591583 26579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:13.591605 26579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:13.591634 26579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:13.593321 26579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:13.593638 26579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:13.593881 26579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:13.598990 26585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:13.599007 26587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:13.598997 26584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:13.599356 26579 server_base.cc:1034] running on GCE node
I20260620 14:55:13.599673 26579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:13.600003 26579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:13.601179 26579 hybrid_clock.cc:648] HybridClock initialized: now 1781967313601160 us; error 40 us; skew 500 ppm
Jun 20 14:55:13 dist-test-slave-2m5f krb5kdc[26566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:13.603842 26579 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:13.604380 26579 webserver.cc:460] Webserver started at http://127.0.0.1:33475/ using document root <none> and password file <none>
I20260620 14:55:13.604598 26579 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:13.604683 26579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:13.604836 26579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:13.605930 26579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "babf57c3f43c4a34b525d5c336724bdb"
format_stamp: "Formatted at 2026-06-20 14:55:13 on dist-test-slave-2m5f"
server_key: "a680d2378f866d01415e12b832314162"
server_key_iv: "1a2f7585626430d0964f851ff0958ef6"
server_key_version: "encryptionkey@0"
I20260620 14:55:13.606237 26579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "babf57c3f43c4a34b525d5c336724bdb"
format_stamp: "Formatted at 2026-06-20 14:55:13 on dist-test-slave-2m5f"
server_key: "a680d2378f866d01415e12b832314162"
server_key_iv: "1a2f7585626430d0964f851ff0958ef6"
server_key_version: "encryptionkey@0"
I20260620 14:55:13.607699 26579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:55:13.608436 26594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:13.608592 26579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:55:13.608693 26579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "babf57c3f43c4a34b525d5c336724bdb"
format_stamp: "Formatted at 2026-06-20 14:55:13 on dist-test-slave-2m5f"
server_key: "a680d2378f866d01415e12b832314162"
server_key_iv: "1a2f7585626430d0964f851ff0958ef6"
server_key_version: "encryptionkey@0"
I20260620 14:55:13.608781 26579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:13.617605 26579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:13.620042 26579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:13.620265 26579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:13.624428 26579 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34381
I20260620 14:55:13.624460 26646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34381 every 8 connection(s)
I20260620 14:55:13.624840 26579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:13.625845 26647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:13.628134 26647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb: Bootstrap starting.
I20260620 14:55:13.628770 26647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:13.629006 26647 log.cc:824] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:13.629725 26647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb: No bootstrap required, opened a new log
I20260620 14:55:13.631347 26647 raft_consensus.cc:348] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf57c3f43c4a34b525d5c336724bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260620 14:55:13.631495 26647 raft_consensus.cc:374] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:13.631518 26647 raft_consensus.cc:729] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babf57c3f43c4a34b525d5c336724bdb, State: Initialized, Role: FOLLOWER
I20260620 14:55:13.631673 26647 consensus_queue.cc:260] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf57c3f43c4a34b525d5c336724bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260620 14:55:13.631755 26647 raft_consensus.cc:388] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:13.631779 26647 raft_consensus.cc:482] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:13.631807 26647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:13.632391 26647 raft_consensus.cc:504] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf57c3f43c4a34b525d5c336724bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260620 14:55:13.632535 26647 leader_election.cc:302] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: babf57c3f43c4a34b525d5c336724bdb; no voters:
I20260620 14:55:13.632684 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 26579
I20260620 14:55:13.632738 26647 leader_election.cc:288] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:13.632805 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:13.632794 26650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:13.632974 26650 raft_consensus.cc:686] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [term 1 LEADER]: Becoming Leader. State: Replica: babf57c3f43c4a34b525d5c336724bdb, State: Running, Role: LEADER
I20260620 14:55:13.633028 5594 external_mini_cluster.cc:1428] Setting key 8caaf81da5ac472b6b743892181b6b48
I20260620 14:55:13.633018 26647 sys_catalog.cc:564] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:13.633072 26650 consensus_queue.cc:237] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf57c3f43c4a34b525d5c336724bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } }
I20260620 14:55:13.633385 26650 sys_catalog.cc:455] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader babf57c3f43c4a34b525d5c336724bdb. Latest consensus state: current_term: 1 leader_uuid: "babf57c3f43c4a34b525d5c336724bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf57c3f43c4a34b525d5c336724bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } } }
I20260620 14:55:13.633471 26650 sys_catalog.cc:458] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:13.633631 26652 sys_catalog.cc:455] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "babf57c3f43c4a34b525d5c336724bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "babf57c3f43c4a34b525d5c336724bdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34381 } } }
I20260620 14:55:13.633744 26652 sys_catalog.cc:458] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:13.634528 26657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:13.635195 26657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:13.637377 26657 catalog_manager.cc:1347] Generated new cluster ID: c32f9991195543bb8ec4574a9922337b
I20260620 14:55:13.637454 26657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:13.639175 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:13.633712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:57425 (local address 127.0.0.1:34381)
0620 14:55:13.633920 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:13.633928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:13.633955 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:55:13.634053 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:13.634058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:13.634159 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:13.634314 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:13.634567 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:13.635952 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:13.636843 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:13.637484 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:13.637603 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:13.638390 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:13.638394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:13.638395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:13.638615 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:13.638664 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:13.638679 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:13.638892 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:13.638943 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:55:13.638970 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260620 14:55:13.640262 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:13.645610 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:13.641683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:36865 (local address 127.0.0.1:34381)
0620 14:55:13.641715 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:13.641717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:13.641827 (+ 110us) server_negotiation.cc:407] Connection header received
0620 14:55:13.641885 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:13.641885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:13.641895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:13.641951 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:13.642083 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:13.643169 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:13.644237 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:13.644564 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:13.644624 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:13.645346 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:13.645347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:13.645347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:13.645351 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:13.645365 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:13.645370 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:13.645423 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:13.645519 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:55:13.645532 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 14:55:13.649807 26657 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:13.650362 26657 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:13.663272 26657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P babf57c3f43c4a34b525d5c336724bdb: Generated new TSK 0
I20260620 14:55:13.663533 26657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:13.670877 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 26579
W20260620 14:55:13.675405 26564 connection.cc:535] client connection to 127.0.0.1:34381 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34381): Connection reset by peer (error 104)
2026-06-20T14:55:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:13 dist-test-slave-2m5f krb5kdc[26678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:13 dist-test-slave-2m5f krb5kdc[26678](info): set up 2 sockets
Jun 20 14:55:13 dist-test-slave-2m5f krb5kdc[26678](info): commencing operation
krb5kdc: starting...
W20260620 14:55:14.724507 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:14 dist-test-slave-2m5f krb5kdc[26678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:14.850850 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:14.936038 26691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:14.936340 26691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:14.936384 26691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:14.937613 26691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:14.937695 26691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:14.937719 26691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:14.937752 26691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:14.937801 26691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:14.939462 26691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:14.939792 26691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:14.940043 26691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:14.945367 26696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:14.945379 26699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:14.945454 26697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:14.945708 26691 server_base.cc:1034] running on GCE node
I20260620 14:55:14.945943 26691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:14.946261 26691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:14.947422 26691 hybrid_clock.cc:648] HybridClock initialized: now 1781967314947401 us; error 31 us; skew 500 ppm
Jun 20 14:55:14 dist-test-slave-2m5f krb5kdc[26678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:14.950275 26691 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:14.950825 26691 webserver.cc:460] Webserver started at http://127.0.0.1:43789/ using document root <none> and password file <none>
I20260620 14:55:14.951076 26691 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:14.951148 26691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:14.951306 26691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:14.952535 26691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "06aef1cdd410495cbcc8298874b90f52"
format_stamp: "Formatted at 2026-06-20 14:55:14 on dist-test-slave-2m5f"
server_key: "04a88dc4e5d76674a76667cd90414e1a"
server_key_iv: "b12b71f20be459b82977b9b93da267f4"
server_key_version: "encryptionkey@0"
I20260620 14:55:14.952944 26691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "06aef1cdd410495cbcc8298874b90f52"
format_stamp: "Formatted at 2026-06-20 14:55:14 on dist-test-slave-2m5f"
server_key: "04a88dc4e5d76674a76667cd90414e1a"
server_key_iv: "b12b71f20be459b82977b9b93da267f4"
server_key_version: "encryptionkey@0"
I20260620 14:55:14.954449 26691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:55:14.955420 26706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:14.955835 26691 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:14.955951 26691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "06aef1cdd410495cbcc8298874b90f52"
format_stamp: "Formatted at 2026-06-20 14:55:14 on dist-test-slave-2m5f"
server_key: "04a88dc4e5d76674a76667cd90414e1a"
server_key_iv: "b12b71f20be459b82977b9b93da267f4"
server_key_version: "encryptionkey@0"
I20260620 14:55:14.956041 26691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:14.959677 26691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:14.962208 26691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:14.962397 26691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:14.966969 26691 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37687
I20260620 14:55:14.967136 26758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37687 every 8 connection(s)
I20260620 14:55:14.967427 26691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:14.968621 26759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:14.971153 26759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52: Bootstrap starting.
I20260620 14:55:14.971797 26759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:14.972105 26759 log.cc:824] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:14.973006 26759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52: No bootstrap required, opened a new log
I20260620 14:55:14.974750 26759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aef1cdd410495cbcc8298874b90f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37687 } }
I20260620 14:55:14.974906 26759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:14.974929 26759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06aef1cdd410495cbcc8298874b90f52, State: Initialized, Role: FOLLOWER
I20260620 14:55:14.975102 26759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aef1cdd410495cbcc8298874b90f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37687 } }
I20260620 14:55:14.975167 26759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:14.975189 26759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:14.975255 26759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:14.976279 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 26691
I20260620 14:55:14.976305 26759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aef1cdd410495cbcc8298874b90f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37687 } }
I20260620 14:55:14.976416 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:14.976419 26759 leader_election.cc:302] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06aef1cdd410495cbcc8298874b90f52; no voters:
I20260620 14:55:14.976614 26759 leader_election.cc:288] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:14.976655 5594 external_mini_cluster.cc:1428] Setting key 2e82a7eecffd4c5e8d4c4de7ba6b6430
I20260620 14:55:14.976755 26762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:14.976908 26759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:14.976969 26762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [term 1 LEADER]: Becoming Leader. State: Replica: 06aef1cdd410495cbcc8298874b90f52, State: Running, Role: LEADER
I20260620 14:55:14.977162 26762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aef1cdd410495cbcc8298874b90f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37687 } }
I20260620 14:55:14.978048 26763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06aef1cdd410495cbcc8298874b90f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aef1cdd410495cbcc8298874b90f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37687 } } }
I20260620 14:55:14.978147 26763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:14.978943 26764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06aef1cdd410495cbcc8298874b90f52. Latest consensus state: current_term: 1 leader_uuid: "06aef1cdd410495cbcc8298874b90f52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06aef1cdd410495cbcc8298874b90f52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37687 } } }
I20260620 14:55:14.979023 26764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52 [sys.catalog]: This master's current role is: LEADER
W20260620 14:55:14.979501 26781 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:55:14.979571 26781 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:55:14.979624 26776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:14.979763 26776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:14.979828 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:14.977313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:43413 (local address 127.0.0.1:37687)
0620 14:55:14.977522 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:14.977534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:14.977592 (+ 58us) server_negotiation.cc:407] Connection header received
0620 14:55:14.977741 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:14.977749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:14.977786 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:14.978855 (+ 1069us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:14.979194 (+ 339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:14.979198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:14.979198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:14.979202 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:14.979245 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:14.979262 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:14.979348 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:14.979468 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 14:55:14.979641 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260620 14:55:14.981000 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:14.981495 26776 catalog_manager.cc:1347] Generated new cluster ID: a48461e075624ff2bf60ba556aeab372
I20260620 14:55:14.981554 26776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:14.985517 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:14.982569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:44701 (local address 127.0.0.1:37687)
0620 14:55:14.982609 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:14.982611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:14.984483 (+ 1872us) server_negotiation.cc:407] Connection header received
0620 14:55:14.984522 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:14.984523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:14.984530 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:14.984601 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:14.985262 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:14.985263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:14.985263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:14.985268 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:14.985286 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:14.985291 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:14.985359 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:14.985413 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 14:55:14.985427 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:55:14.991341 26776 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:14.991549 26776 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:15.002526 26776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06aef1cdd410495cbcc8298874b90f52: Generated new TSK 0
I20260620 14:55:15.002777 26776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:15.006029 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 26691
2026-06-20T14:55:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:15 dist-test-slave-2m5f krb5kdc[26791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:15 dist-test-slave-2m5f krb5kdc[26791](info): set up 2 sockets
Jun 20 14:55:15 dist-test-slave-2m5f krb5kdc[26791](info): commencing operation
krb5kdc: starting...
W20260620 14:55:16.064813 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:16 dist-test-slave-2m5f krb5kdc[26791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:16.200017 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:16.283159 26804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:16.283440 26804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:16.283484 26804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:16.284758 26804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:16.284845 26804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:16.284871 26804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:16.284901 26804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:16.284933 26804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:16.286525 26804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43521
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:16.286842 26804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:16.287107 26804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:16.292505 26810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:16.293166 26809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:16.293166 26812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:16.293435 26804 server_base.cc:1034] running on GCE node
I20260620 14:55:16.293778 26804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:16.294106 26804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:16.295281 26804 hybrid_clock.cc:648] HybridClock initialized: now 1781967316295258 us; error 37 us; skew 500 ppm
Jun 20 14:55:16 dist-test-slave-2m5f krb5kdc[26791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:16.297999 26804 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:16.298465 26804 webserver.cc:460] Webserver started at http://127.0.0.1:36117/ using document root <none> and password file <none>
I20260620 14:55:16.298722 26804 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:16.298804 26804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:16.298956 26804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:16.300066 26804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "8382a3cf9b5043c0adad1ee7c08418ed"
format_stamp: "Formatted at 2026-06-20 14:55:16 on dist-test-slave-2m5f"
server_key: "97568b1e601d7eae93a8be9ccc273aa5"
server_key_iv: "2e78ac5b6d74b19b30165021b5c3e586"
server_key_version: "encryptionkey@0"
I20260620 14:55:16.300493 26804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "8382a3cf9b5043c0adad1ee7c08418ed"
format_stamp: "Formatted at 2026-06-20 14:55:16 on dist-test-slave-2m5f"
server_key: "97568b1e601d7eae93a8be9ccc273aa5"
server_key_iv: "2e78ac5b6d74b19b30165021b5c3e586"
server_key_version: "encryptionkey@0"
I20260620 14:55:16.301915 26804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:16.302778 26819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:16.302996 26804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:16.303104 26804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "8382a3cf9b5043c0adad1ee7c08418ed"
format_stamp: "Formatted at 2026-06-20 14:55:16 on dist-test-slave-2m5f"
server_key: "97568b1e601d7eae93a8be9ccc273aa5"
server_key_iv: "2e78ac5b6d74b19b30165021b5c3e586"
server_key_version: "encryptionkey@0"
I20260620 14:55:16.303191 26804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:16.319988 26804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:16.322211 26804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:16.322394 26804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:16.326813 26804 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43521
I20260620 14:55:16.326848 26871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43521 every 8 connection(s)
I20260620 14:55:16.327220 26804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:16.328444 26872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:16.330914 26872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed: Bootstrap starting.
I20260620 14:55:16.331517 26872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:16.331789 26872 log.cc:824] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:16.332594 26872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed: No bootstrap required, opened a new log
I20260620 14:55:16.334432 26872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8382a3cf9b5043c0adad1ee7c08418ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43521 } }
I20260620 14:55:16.334609 26872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:16.334635 26872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8382a3cf9b5043c0adad1ee7c08418ed, State: Initialized, Role: FOLLOWER
I20260620 14:55:16.334816 26872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8382a3cf9b5043c0adad1ee7c08418ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43521 } }
I20260620 14:55:16.334910 26872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:16.334937 26872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:16.334969 26872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:16.335383 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 26804
I20260620 14:55:16.335507 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:16.335716 5594 external_mini_cluster.cc:1428] Setting key bd7ca1344a375484b98294b6e60d108f
I20260620 14:55:16.335850 26872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8382a3cf9b5043c0adad1ee7c08418ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43521 } }
I20260620 14:55:16.336006 26872 leader_election.cc:302] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8382a3cf9b5043c0adad1ee7c08418ed; no voters:
I20260620 14:55:16.336270 26872 leader_election.cc:288] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:16.336483 26877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:16.336611 26872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:16.336733 26877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [term 1 LEADER]: Becoming Leader. State: Replica: 8382a3cf9b5043c0adad1ee7c08418ed, State: Running, Role: LEADER
I20260620 14:55:16.336870 26877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8382a3cf9b5043c0adad1ee7c08418ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43521 } }
I20260620 14:55:16.337466 26875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8382a3cf9b5043c0adad1ee7c08418ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8382a3cf9b5043c0adad1ee7c08418ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43521 } } }
I20260620 14:55:16.337563 26875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:16.337764 26879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8382a3cf9b5043c0adad1ee7c08418ed. Latest consensus state: current_term: 1 leader_uuid: "8382a3cf9b5043c0adad1ee7c08418ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8382a3cf9b5043c0adad1ee7c08418ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43521 } } }
I20260620 14:55:16.337925 26879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed [sys.catalog]: This master's current role is: LEADER
W20260620 14:55:16.338622 26892 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:55:16.338685 26892 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:55:16.338742 26886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:16.338901 26886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:16.340745 26886 catalog_manager.cc:1347] Generated new cluster ID: d11105bafb674bcba5a77e3518c0e0f6
I20260620 14:55:16.340827 26886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:55:16 dist-test-slave-2m5f krb5kdc[26791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:55:16.349556 26878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:16.336361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:54295 (local address 127.0.0.1:43521)
0620 14:55:16.336680 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:16.336692 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:16.336750 (+ 58us) server_negotiation.cc:407] Connection header received
0620 14:55:16.336886 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:16.336894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:16.336960 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:16.338014 (+ 1054us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:16.338142 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:16.339573 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:16.340684 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:16.341391 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:16.341527 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:16.344651 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:16.344657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:16.344657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:16.344671 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:16.347439 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:16.348001 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:16.348005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:16.348005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:16.348144 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:16.348448 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:16.348449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:16.348449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:16.348779 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:16.348930 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:16.349099 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 14:55:16.349321 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":70,"threads_started":1}
I20260620 14:55:16.350580 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:16.360136 26886 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:16.360529 26878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:16.352250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:56985 (local address 127.0.0.1:43521)
0620 14:55:16.352318 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:16.352320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:16.352444 (+ 124us) server_negotiation.cc:407] Connection header received
0620 14:55:16.352493 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:16.352494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:16.352505 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:16.352564 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:16.353187 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:16.354060 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:16.355047 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:16.355426 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:16.355507 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:16.357122 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:16.357123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:16.357124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:16.357132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:16.358909 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:16.359358 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:16.359359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:16.359359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:16.359427 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:16.359708 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:16.359709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:16.359709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:16.359868 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:16.360002 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:16.360349 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 14:55:16.360427 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 14:55:16.361189 26886 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:16.365934 26886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8382a3cf9b5043c0adad1ee7c08418ed: Generated new TSK 0
I20260620 14:55:16.366173 26886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:16.371114 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 26804
W20260620 14:55:16.375381 26789 connection.cc:535] client connection to 127.0.0.1:43521 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43521): Connection reset by peer (error 104)
2026-06-20T14:55:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:16 dist-test-slave-2m5f krb5kdc[26903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:16 dist-test-slave-2m5f krb5kdc[26903](info): set up 2 sockets
Jun 20 14:55:16 dist-test-slave-2m5f krb5kdc[26903](info): commencing operation
krb5kdc: starting...
W20260620 14:55:17.420737 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:17 dist-test-slave-2m5f krb5kdc[26903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:17.545197 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:17.628526 26916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:17.628767 26916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:17.628790 26916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:17.630018 26916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:17.630066 26916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:17.630080 26916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:17.630093 26916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:17.630106 26916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:17.631649 26916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46173
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:17.631927 26916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:17.632205 26916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:17.637384 26924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:17.637403 26921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:17.637610 26916 server_base.cc:1034] running on GCE node
W20260620 14:55:17.637403 26922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:17.638070 26916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:17.638419 26916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:17.639611 26916 hybrid_clock.cc:648] HybridClock initialized: now 1781967317639595 us; error 27 us; skew 500 ppm
Jun 20 14:55:17 dist-test-slave-2m5f krb5kdc[26903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:17.645987 26916 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:17.646476 26916 webserver.cc:460] Webserver started at http://127.0.0.1:43381/ using document root <none> and password file <none>
I20260620 14:55:17.646715 26916 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:17.646791 26916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:17.646943 26916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:17.648114 26916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "67bab0d722174da190fb3a63d0919832"
format_stamp: "Formatted at 2026-06-20 14:55:17 on dist-test-slave-2m5f"
server_key: "949fd7bb4c562070d849592b284bc209"
server_key_iv: "c584edd6e0e4abc21b83c652f787e4f6"
server_key_version: "encryptionkey@0"
I20260620 14:55:17.648566 26916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "67bab0d722174da190fb3a63d0919832"
format_stamp: "Formatted at 2026-06-20 14:55:17 on dist-test-slave-2m5f"
server_key: "949fd7bb4c562070d849592b284bc209"
server_key_iv: "c584edd6e0e4abc21b83c652f787e4f6"
server_key_version: "encryptionkey@0"
I20260620 14:55:17.650007 26916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:17.650851 26931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:17.651124 26916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:17.651374 26916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "67bab0d722174da190fb3a63d0919832"
format_stamp: "Formatted at 2026-06-20 14:55:17 on dist-test-slave-2m5f"
server_key: "949fd7bb4c562070d849592b284bc209"
server_key_iv: "c584edd6e0e4abc21b83c652f787e4f6"
server_key_version: "encryptionkey@0"
I20260620 14:55:17.651463 26916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:17.679105 26916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:17.681375 26916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:17.681550 26916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:17.685794 26916 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46173
I20260620 14:55:17.685833 26983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46173 every 8 connection(s)
I20260620 14:55:17.686214 26916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:17.687180 26984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:17.689607 26984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832: Bootstrap starting.
I20260620 14:55:17.690155 26984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:17.690424 26984 log.cc:824] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:17.690626 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 26916
I20260620 14:55:17.690800 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:17.691004 5594 external_mini_cluster.cc:1428] Setting key beb5fd91667c0a5af26373010261e823
I20260620 14:55:17.691267 26984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832: No bootstrap required, opened a new log
I20260620 14:55:17.693277 26984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab0d722174da190fb3a63d0919832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260620 14:55:17.693454 26984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:17.693500 26984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bab0d722174da190fb3a63d0919832, State: Initialized, Role: FOLLOWER
I20260620 14:55:17.693629 26984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab0d722174da190fb3a63d0919832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260620 14:55:17.693732 26984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:17.693778 26984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:17.693835 26984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:17.694420 26984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab0d722174da190fb3a63d0919832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260620 14:55:17.694545 26984 leader_election.cc:302] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67bab0d722174da190fb3a63d0919832; no voters:
I20260620 14:55:17.694728 26984 leader_election.cc:288] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:17.694777 26989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:17.694931 26989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [term 1 LEADER]: Becoming Leader. State: Replica: 67bab0d722174da190fb3a63d0919832, State: Running, Role: LEADER
I20260620 14:55:17.695010 26984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:17.695016 26989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab0d722174da190fb3a63d0919832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } }
I20260620 14:55:17.695397 26989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67bab0d722174da190fb3a63d0919832. Latest consensus state: current_term: 1 leader_uuid: "67bab0d722174da190fb3a63d0919832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab0d722174da190fb3a63d0919832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } } }
I20260620 14:55:17.695410 26991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67bab0d722174da190fb3a63d0919832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67bab0d722174da190fb3a63d0919832" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46173 } } }
I20260620 14:55:17.695489 26989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:17.695518 26991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832 [sys.catalog]: This master's current role is: LEADER
W20260620 14:55:17.697093 27005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:55:17.697149 27005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:55:17.697255 26995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:17.697497 26995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:17.699016 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:17.691562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:40303 (local address 127.0.0.1:46173)
0620 14:55:17.691771 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:17.691782 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:17.691814 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:55:17.692579 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:17.692587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:17.692643 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:17.693445 (+ 802us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:17.693687 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:17.695156 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:17.696441 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:17.697169 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:17.697329 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:17.698063 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:17.698069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:17.698069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:17.698285 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:17.698319 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:17.698331 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:17.698431 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:17.698518 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:55:17.698780 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260620 14:55:17.699304 26995 catalog_manager.cc:1347] Generated new cluster ID: 808cfc81990547c8a0fcf825142cdb41
I20260620 14:55:17.699365 26995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:17.700035 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:17.707746 26988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:17.704593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:52775 (local address 127.0.0.1:46173)
0620 14:55:17.704635 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:17.704637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:17.704816 (+ 179us) server_negotiation.cc:407] Connection header received
0620 14:55:17.704889 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:17.704890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:17.704902 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:17.704952 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:17.705101 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:17.705587 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:17.706325 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:17.706546 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:17.706634 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:17.707376 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:17.707377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:17.707378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:17.707383 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:17.707398 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:17.707403 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:17.707481 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:17.707623 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:55:17.707635 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 14:55:17.710227 26995 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:17.710966 26995 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:17.725303 26995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67bab0d722174da190fb3a63d0919832: Generated new TSK 0
I20260620 14:55:17.726158 26995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:17.735457 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 26916
W20260620 14:55:17.740110 26901 connection.cc:535] client connection to 127.0.0.1:46173 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46173): Connection reset by peer (error 104)
2026-06-20T14:55:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:17 dist-test-slave-2m5f krb5kdc[27015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:17 dist-test-slave-2m5f krb5kdc[27015](info): set up 2 sockets
Jun 20 14:55:17 dist-test-slave-2m5f krb5kdc[27015](info): commencing operation
krb5kdc: starting...
W20260620 14:55:18.780530 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:18 dist-test-slave-2m5f krb5kdc[27015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:55:18.907754 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:18.995980 27028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:18.996232 27028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:18.996256 27028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:18.997520 27028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:18.997570 27028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:18.997583 27028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:18.997598 27028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:18.997611 27028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:18.999157 27028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:18.999414 27028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:18.999670 27028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:19.004763 27034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:19.004746 27036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:19.004742 27033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:19.005138 27028 server_base.cc:1034] running on GCE node
I20260620 14:55:19.005425 27028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:19.005762 27028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:19.006956 27028 hybrid_clock.cc:648] HybridClock initialized: now 1781967319006932 us; error 37 us; skew 500 ppm
Jun 20 14:55:19 dist-test-slave-2m5f krb5kdc[27015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:19.009790 27028 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:55:19.010293 27028 webserver.cc:460] Webserver started at http://127.0.0.1:45045/ using document root <none> and password file <none>
I20260620 14:55:19.010510 27028 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:19.010551 27028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:19.010643 27028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:19.011703 27028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "102acdfdd58e44298761173ff5e1c64c"
format_stamp: "Formatted at 2026-06-20 14:55:19 on dist-test-slave-2m5f"
server_key: "22b6f1bed28fb66b53e6187201913f18"
server_key_iv: "851d25d2893c0eed37c0f5f46fb3a3b6"
server_key_version: "encryptionkey@0"
I20260620 14:55:19.012014 27028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "102acdfdd58e44298761173ff5e1c64c"
format_stamp: "Formatted at 2026-06-20 14:55:19 on dist-test-slave-2m5f"
server_key: "22b6f1bed28fb66b53e6187201913f18"
server_key_iv: "851d25d2893c0eed37c0f5f46fb3a3b6"
server_key_version: "encryptionkey@0"
I20260620 14:55:19.013479 27028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:55:19.014360 27043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:19.014562 27028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:19.014650 27028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "102acdfdd58e44298761173ff5e1c64c"
format_stamp: "Formatted at 2026-06-20 14:55:19 on dist-test-slave-2m5f"
server_key: "22b6f1bed28fb66b53e6187201913f18"
server_key_iv: "851d25d2893c0eed37c0f5f46fb3a3b6"
server_key_version: "encryptionkey@0"
I20260620 14:55:19.014708 27028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:19.018108 27028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:19.020732 27028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:19.020915 27028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:19.025378 27028 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32795
I20260620 14:55:19.025411 27095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32795 every 8 connection(s)
I20260620 14:55:19.025784 27028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:19.026795 27096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:19.029209 27096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c: Bootstrap starting.
I20260620 14:55:19.029801 27096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:19.030094 27096 log.cc:824] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:19.030817 27096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c: No bootstrap required, opened a new log
I20260620 14:55:19.032603 27096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102acdfdd58e44298761173ff5e1c64c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32795 } }
I20260620 14:55:19.032757 27096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:19.032816 27096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102acdfdd58e44298761173ff5e1c64c, State: Initialized, Role: FOLLOWER
I20260620 14:55:19.032998 27096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102acdfdd58e44298761173ff5e1c64c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32795 } }
I20260620 14:55:19.033102 27096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:19.033141 27096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:19.033150 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 27028
I20260620 14:55:19.033193 27096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:19.033264 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:19.033479 5594 external_mini_cluster.cc:1428] Setting key 089cdb94f8a59c4179cc32582bbb1532
I20260620 14:55:19.033910 27096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102acdfdd58e44298761173ff5e1c64c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32795 } }
I20260620 14:55:19.034020 27096 leader_election.cc:302] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 102acdfdd58e44298761173ff5e1c64c; no voters:
I20260620 14:55:19.034189 27096 leader_election.cc:288] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:19.034214 27101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:19.034356 27101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [term 1 LEADER]: Becoming Leader. State: Replica: 102acdfdd58e44298761173ff5e1c64c, State: Running, Role: LEADER
I20260620 14:55:19.034452 27096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:19.034453 27101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102acdfdd58e44298761173ff5e1c64c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32795 } }
I20260620 14:55:19.034827 27103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 102acdfdd58e44298761173ff5e1c64c. Latest consensus state: current_term: 1 leader_uuid: "102acdfdd58e44298761173ff5e1c64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102acdfdd58e44298761173ff5e1c64c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32795 } } }
I20260620 14:55:19.034823 27102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "102acdfdd58e44298761173ff5e1c64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "102acdfdd58e44298761173ff5e1c64c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32795 } } }
I20260620 14:55:19.034932 27102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:19.035001 27103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:19.035209 27105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:19.036398 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:19.033971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:39845 (local address 127.0.0.1:32795)
0620 14:55:19.034155 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:19.034165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:19.034363 (+ 198us) server_negotiation.cc:407] Connection header received
0620 14:55:19.034500 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:19.034508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:19.034541 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:19.034718 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:19.035863 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:19.035867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:19.035868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:19.035872 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:19.035924 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:19.035942 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:19.036018 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:19.036127 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 14:55:19.036211 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260620 14:55:19.036402 27105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:19.037469 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:19.038230 27105 catalog_manager.cc:1347] Generated new cluster ID: f23f2c0aa37449d5bcf415b3c37606e2
I20260620 14:55:19.038285 27105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:19.040063 27100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:19.038929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.243:50903 (local address 127.0.0.1:32795)
0620 14:55:19.038963 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:19.038965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:19.039101 (+ 136us) server_negotiation.cc:407] Connection header received
0620 14:55:19.039145 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:19.039146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:19.039151 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:19.039212 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:19.039829 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:19.039830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:19.039830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:19.039835 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:19.039851 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:19.039856 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:19.039920 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:19.039975 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:55:19.039989 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:55:19.045641 27105 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:19.045847 27105 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:19.048944 27105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 102acdfdd58e44298761173ff5e1c64c: Generated new TSK 0
I20260620 14:55:19.049139 27105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:19.055809 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 27028
2026-06-20T14:55:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:19 dist-test-slave-2m5f krb5kdc[27127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:19 dist-test-slave-2m5f krb5kdc[27127](info): set up 2 sockets
Jun 20 14:55:19 dist-test-slave-2m5f krb5kdc[27127](info): commencing operation
krb5kdc: starting...
W20260620 14:55:20.120429 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:20 dist-test-slave-2m5f krb5kdc[27127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:55:20.247506 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:37055
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:37055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:20.330622 27140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:20.330843 27140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:20.330868 27140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:20.332007 27140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:20.332048 27140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:20.332062 27140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:20.332074 27140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:20.332096 27140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:20.333699 27140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:37055
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:37055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:20.333938 27140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:20.334117 27140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:20.339213 27148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:20.339236 27145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:20.339213 27146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:20.339574 27140 server_base.cc:1034] running on GCE node
I20260620 14:55:20.339820 27140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:20.340130 27140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:20.341329 27140 hybrid_clock.cc:648] HybridClock initialized: now 1781967320341308 us; error 40 us; skew 500 ppm
Jun 20 14:55:20 dist-test-slave-2m5f krb5kdc[27127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967320, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:20.343897 27140 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:55:20.344446 27140 webserver.cc:460] Webserver started at http://127.5.118.190:37141/ using document root <none> and password file <none>
I20260620 14:55:20.344650 27140 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:20.344691 27140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:20.344782 27140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:20.345834 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "f3f1e38a1f4e4291ba1de2264b6f78da"
format_stamp: "Formatted at 2026-06-20 14:55:20 on dist-test-slave-2m5f"
server_key: "da182d766c930320a3a174493b6d5f4a"
server_key_iv: "d365089a7d7e2be4f28c8cb4279f2c89"
server_key_version: "encryptionkey@0"
I20260620 14:55:20.346154 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "f3f1e38a1f4e4291ba1de2264b6f78da"
format_stamp: "Formatted at 2026-06-20 14:55:20 on dist-test-slave-2m5f"
server_key: "da182d766c930320a3a174493b6d5f4a"
server_key_iv: "d365089a7d7e2be4f28c8cb4279f2c89"
server_key_version: "encryptionkey@0"
I20260620 14:55:20.347592 27140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:55:20.348451 27155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:20.348626 27140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:20.348712 27140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "f3f1e38a1f4e4291ba1de2264b6f78da"
format_stamp: "Formatted at 2026-06-20 14:55:20 on dist-test-slave-2m5f"
server_key: "da182d766c930320a3a174493b6d5f4a"
server_key_iv: "d365089a7d7e2be4f28c8cb4279f2c89"
server_key_version: "encryptionkey@0"
I20260620 14:55:20.348773 27140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:20.373621 27140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:20.375895 27140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:20.376030 27140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:20.380239 27140 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:37055
I20260620 14:55:20.380402 27207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:37055 every 8 connection(s)
I20260620 14:55:20.380731 27140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:20.381702 27208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:20.382936 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 27140
I20260620 14:55:20.383069 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:20.383253 5594 external_mini_cluster.cc:1428] Setting key f032075c46b9290a898b5e6311477560
I20260620 14:55:20.384995 27208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da: Bootstrap starting.
I20260620 14:55:20.385675 27208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:20.386024 27208 log.cc:824] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:20.386858 27208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da: No bootstrap required, opened a new log
I20260620 14:55:20.389145 27208 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f1e38a1f4e4291ba1de2264b6f78da" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37055 } }
I20260620 14:55:20.389320 27208 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:20.389359 27208 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f1e38a1f4e4291ba1de2264b6f78da, State: Initialized, Role: FOLLOWER
I20260620 14:55:20.389520 27208 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f1e38a1f4e4291ba1de2264b6f78da" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37055 } }
I20260620 14:55:20.389609 27208 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:20.389654 27208 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:20.389707 27208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:20.390295 27208 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f1e38a1f4e4291ba1de2264b6f78da" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37055 } }
I20260620 14:55:20.390424 27208 leader_election.cc:302] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3f1e38a1f4e4291ba1de2264b6f78da; no voters:
I20260620 14:55:20.390647 27208 leader_election.cc:288] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:20.390669 27213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:20.390797 27213 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [term 1 LEADER]: Becoming Leader. State: Replica: f3f1e38a1f4e4291ba1de2264b6f78da, State: Running, Role: LEADER
I20260620 14:55:20.390957 27208 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 14:55:20 dist-test-slave-2m5f krb5kdc[27127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:20.390942 27213 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f1e38a1f4e4291ba1de2264b6f78da" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37055 } }
I20260620 14:55:20.391409 27215 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3f1e38a1f4e4291ba1de2264b6f78da. Latest consensus state: current_term: 1 leader_uuid: "f3f1e38a1f4e4291ba1de2264b6f78da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f1e38a1f4e4291ba1de2264b6f78da" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37055 } } }
I20260620 14:55:20.391499 27215 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:20.392035 27214 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3f1e38a1f4e4291ba1de2264b6f78da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f1e38a1f4e4291ba1de2264b6f78da" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37055 } } }
I20260620 14:55:20.392159 27214 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:20.392432 27220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:20.392985 27220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:20.394878 27220 catalog_manager.cc:1347] Generated new cluster ID: 291d6ecaf936438490c8778d289bcc49
I20260620 14:55:20.394927 27220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:20.395884 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:20.383834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33775 (local address 127.5.118.190:37055)
0620 14:55:20.384032 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:20.384043 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:20.384077 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:55:20.384655 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:20.384665 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:20.384733 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:20.384893 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:20.385630 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:20.387028 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:20.387841 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:20.388571 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:20.388711 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:20.391712 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:20.391717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:20.391717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:20.391732 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:20.393971 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:20.394490 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:20.394493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:20.394494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:20.394601 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:20.394908 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:20.394909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:20.394910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:20.395223 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:20.395384 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:20.395547 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:55:20.395656 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260620 14:55:20.396680 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:20.407858 27211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:20.398091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36581 (local address 127.5.118.190:37055)
0620 14:55:20.398124 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:20.398125 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:20.400615 (+ 2490us) server_negotiation.cc:407] Connection header received
0620 14:55:20.400650 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:20.400651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:20.400663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:20.400732 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:20.401337 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:20.402130 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:20.403130 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:20.403497 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:20.403570 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:20.404988 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:20.404989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:20.404990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:20.404997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:20.406619 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:20.407037 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:20.407038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:20.407038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:20.407100 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:20.407326 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:20.407327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:20.407327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:20.407469 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:20.407577 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:20.407692 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:55:20.407758 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 14:55:20.442755 27220 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:20.443549 27220 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:20.457067 27220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3f1e38a1f4e4291ba1de2264b6f78da: Generated new TSK 0
I20260620 14:55:20.457283 27220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:20.460309 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 27140
W20260620 14:55:20.465161 27125 connection.cc:535] client connection to 127.5.118.190:37055 recv error: Network error: failed to read from TLS socket (remote: 127.5.118.190:37055): Connection reset by peer (error 104)
2026-06-20T14:55:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:20 dist-test-slave-2m5f krb5kdc[27239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:20 dist-test-slave-2m5f krb5kdc[27239](info): set up 2 sockets
Jun 20 14:55:20 dist-test-slave-2m5f krb5kdc[27239](info): commencing operation
krb5kdc: starting...
W20260620 14:55:21.508467 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:21 dist-test-slave-2m5f krb5kdc[27239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:55:21.634976 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:44871
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:44871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:21.719648 27252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:21.719920 27252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:21.719962 27252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:21.721248 27252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:21.721333 27252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:21.721357 27252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:21.721376 27252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:21.721402 27252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:21.722945 27252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:44871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:44871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:21.723258 27252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:21.723506 27252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:21.728741 27257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:21.728766 27258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:21.728765 27260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:21.728987 27252 server_base.cc:1034] running on GCE node
I20260620 14:55:21.729307 27252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:21.729633 27252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:21.730815 27252 hybrid_clock.cc:648] HybridClock initialized: now 1781967321730802 us; error 48 us; skew 500 ppm
Jun 20 14:55:21 dist-test-slave-2m5f krb5kdc[27239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967321, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:21.733712 27252 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:55:21.734267 27252 webserver.cc:460] Webserver started at http://127.5.118.190:33135/ using document root <none> and password file <none>
I20260620 14:55:21.734532 27252 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:21.734606 27252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:21.734759 27252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:21.735872 27252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "d6a149e7cd4b46ba9939c1968e8821b3"
format_stamp: "Formatted at 2026-06-20 14:55:21 on dist-test-slave-2m5f"
server_key: "5b6237bb5b4826ed18d703c4d9fe0599"
server_key_iv: "5e15cffa1549e2db19e388950b646dc4"
server_key_version: "encryptionkey@0"
I20260620 14:55:21.736327 27252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "d6a149e7cd4b46ba9939c1968e8821b3"
format_stamp: "Formatted at 2026-06-20 14:55:21 on dist-test-slave-2m5f"
server_key: "5b6237bb5b4826ed18d703c4d9fe0599"
server_key_iv: "5e15cffa1549e2db19e388950b646dc4"
server_key_version: "encryptionkey@0"
I20260620 14:55:21.737738 27252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:55:21.738575 27267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:21.738761 27252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:21.738866 27252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "d6a149e7cd4b46ba9939c1968e8821b3"
format_stamp: "Formatted at 2026-06-20 14:55:21 on dist-test-slave-2m5f"
server_key: "5b6237bb5b4826ed18d703c4d9fe0599"
server_key_iv: "5e15cffa1549e2db19e388950b646dc4"
server_key_version: "encryptionkey@0"
I20260620 14:55:21.738951 27252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:21.753409 27252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:21.755944 27252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:21.756132 27252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:21.760036 27252 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:44871
I20260620 14:55:21.760069 27319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:44871 every 8 connection(s)
I20260620 14:55:21.760520 27252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:21.761561 27320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:21.763904 27320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3: Bootstrap starting.
I20260620 14:55:21.764593 27320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:21.764895 27320 log.cc:824] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:21.765733 27320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3: No bootstrap required, opened a new log
I20260620 14:55:21.767428 27320 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a149e7cd4b46ba9939c1968e8821b3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44871 } }
I20260620 14:55:21.767582 27320 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:21.767606 27320 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a149e7cd4b46ba9939c1968e8821b3, State: Initialized, Role: FOLLOWER
I20260620 14:55:21.767795 27320 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a149e7cd4b46ba9939c1968e8821b3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44871 } }
I20260620 14:55:21.767884 27320 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:21.767940 27320 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:21.767992 27320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:21.768983 27320 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a149e7cd4b46ba9939c1968e8821b3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44871 } }
I20260620 14:55:21.769081 27320 leader_election.cc:302] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a149e7cd4b46ba9939c1968e8821b3; no voters:
I20260620 14:55:21.769285 27320 leader_election.cc:288] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:21.769345 27323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:21.769577 27320 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:21.769584 27323 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [term 1 LEADER]: Becoming Leader. State: Replica: d6a149e7cd4b46ba9939c1968e8821b3, State: Running, Role: LEADER
I20260620 14:55:21.769773 27323 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a149e7cd4b46ba9939c1968e8821b3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44871 } }
I20260620 14:55:21.770187 27325 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a149e7cd4b46ba9939c1968e8821b3. Latest consensus state: current_term: 1 leader_uuid: "d6a149e7cd4b46ba9939c1968e8821b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a149e7cd4b46ba9939c1968e8821b3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44871 } } }
I20260620 14:55:21.770355 27325 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:21.770534 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 27252
I20260620 14:55:21.770664 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:21.770876 5594 external_mini_cluster.cc:1428] Setting key 71481d9171620cc732fd29eef3d42fb3
I20260620 14:55:21.770190 27324 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6a149e7cd4b46ba9939c1968e8821b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a149e7cd4b46ba9939c1968e8821b3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44871 } } }
I20260620 14:55:21.770985 27332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:21.770989 27324 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:21.771669 27332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:21.773469 27332 catalog_manager.cc:1347] Generated new cluster ID: 52760959b0124acdbf7a0b88560ed1f0
I20260620 14:55:21.773531 27332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:21.777212 27341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:21.771380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38647 (local address 127.5.118.190:44871)
0620 14:55:21.771560 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:21.771570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:21.771601 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:55:21.771735 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:21.771743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:21.771809 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:21.771960 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:21.772241 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:21.773672 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:21.774821 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:21.775500 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:21.775650 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:21.776453 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:21.776458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:21.776458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:21.776695 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:21.776737 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:21.776753 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:21.776851 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:21.776938 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:55:21.776969 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260620 14:55:21.778329 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:21.783576 27341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:21.779996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32911 (local address 127.5.118.190:44871)
0620 14:55:21.780042 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:21.780044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:21.780088 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:55:21.780152 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:21.780152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:21.780162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:21.780258 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:21.780373 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:21.781137 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:21.782150 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:21.782477 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:21.782558 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:21.783292 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:21.783294 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:21.783294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:21.783299 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:21.783318 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:21.783323 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:21.783384 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:21.783463 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:55:21.783479 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:55:21.785712 27332 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:21.786428 27332 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:21.792021 27332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6a149e7cd4b46ba9939c1968e8821b3: Generated new TSK 0
I20260620 14:55:21.792279 27332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:21.795222 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 27252
W20260620 14:55:21.799435 27237 connection.cc:535] client connection to 127.5.118.190:44871 recv error: Network error: failed to read from TLS socket (remote: 127.5.118.190:44871): Connection reset by peer (error 104)
2026-06-20T14:55:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:21 dist-test-slave-2m5f krb5kdc[27351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:21 dist-test-slave-2m5f krb5kdc[27351](info): set up 2 sockets
Jun 20 14:55:21 dist-test-slave-2m5f krb5kdc[27351](info): commencing operation
krb5kdc: starting...
W20260620 14:55:22.836896 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:22 dist-test-slave-2m5f krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:55:22.961627 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:46847
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:46847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:23.044576 27364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:23.044799 27364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:23.044824 27364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:23.046065 27364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:23.046118 27364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:23.046132 27364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:23.046151 27364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:23.046164 27364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:23.047685 27364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:46847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:46847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:23.047945 27364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:23.048270 27364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:23.053400 27369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:23.053431 27364 server_base.cc:1034] running on GCE node
W20260620 14:55:23.053412 27370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:23.053412 27372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:23.053812 27364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:23.054164 27364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:23.055339 27364 hybrid_clock.cc:648] HybridClock initialized: now 1781967323055329 us; error 58 us; skew 500 ppm
Jun 20 14:55:23 dist-test-slave-2m5f krb5kdc[27351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967323, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:23.061895 27364 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:55:23.062471 27364 webserver.cc:460] Webserver started at http://127.5.118.190:33329/ using document root <none> and password file <none>
I20260620 14:55:23.062729 27364 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:23.062776 27364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:23.063030 27364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:23.064255 27364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "4a8f5cdb58d74eb6bc748f03373b0d5c"
format_stamp: "Formatted at 2026-06-20 14:55:23 on dist-test-slave-2m5f"
server_key: "11520b1585a282efe07ff7d6e4a590f1"
server_key_iv: "55078c24fb2b6d23a491c045457b334a"
server_key_version: "encryptionkey@0"
I20260620 14:55:23.064656 27364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "4a8f5cdb58d74eb6bc748f03373b0d5c"
format_stamp: "Formatted at 2026-06-20 14:55:23 on dist-test-slave-2m5f"
server_key: "11520b1585a282efe07ff7d6e4a590f1"
server_key_iv: "55078c24fb2b6d23a491c045457b334a"
server_key_version: "encryptionkey@0"
I20260620 14:55:23.066088 27364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:55:23.066936 27379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:23.067227 27364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:23.067332 27364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "4a8f5cdb58d74eb6bc748f03373b0d5c"
format_stamp: "Formatted at 2026-06-20 14:55:23 on dist-test-slave-2m5f"
server_key: "11520b1585a282efe07ff7d6e4a590f1"
server_key_iv: "55078c24fb2b6d23a491c045457b334a"
server_key_version: "encryptionkey@0"
I20260620 14:55:23.067411 27364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:23.071070 27364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:23.073702 27364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:23.073874 27364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:23.078469 27364 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:46847
I20260620 14:55:23.078504 27431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:46847 every 8 connection(s)
I20260620 14:55:23.078866 27364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:23.079954 27432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:23.082521 27432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c: Bootstrap starting.
I20260620 14:55:23.083096 27432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:23.083353 27432 log.cc:824] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:23.084241 27432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c: No bootstrap required, opened a new log
I20260620 14:55:23.085866 27432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8f5cdb58d74eb6bc748f03373b0d5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46847 } }
I20260620 14:55:23.086007 27432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:23.086032 27432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a8f5cdb58d74eb6bc748f03373b0d5c, State: Initialized, Role: FOLLOWER
I20260620 14:55:23.086140 27432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8f5cdb58d74eb6bc748f03373b0d5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46847 } }
I20260620 14:55:23.086226 27432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:23.086261 27432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:23.086337 27432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:23.086717 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 27364
I20260620 14:55:23.086817 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:23.087057 5594 external_mini_cluster.cc:1428] Setting key 3b78213faf88a8c5ca55ddfcce8fbadb
I20260620 14:55:23.087450 27432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8f5cdb58d74eb6bc748f03373b0d5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46847 } }
I20260620 14:55:23.087575 27432 leader_election.cc:302] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a8f5cdb58d74eb6bc748f03373b0d5c; no voters:
I20260620 14:55:23.087771 27432 leader_election.cc:288] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:23.087791 27437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:23.087921 27437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [term 1 LEADER]: Becoming Leader. State: Replica: 4a8f5cdb58d74eb6bc748f03373b0d5c, State: Running, Role: LEADER
I20260620 14:55:23.088032 27432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:23.088088 27437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8f5cdb58d74eb6bc748f03373b0d5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46847 } }
I20260620 14:55:23.088462 27437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a8f5cdb58d74eb6bc748f03373b0d5c. Latest consensus state: current_term: 1 leader_uuid: "4a8f5cdb58d74eb6bc748f03373b0d5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8f5cdb58d74eb6bc748f03373b0d5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46847 } } }
I20260620 14:55:23.088536 27437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:23.088754 27443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:23.089390 27443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:23.089301 27438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a8f5cdb58d74eb6bc748f03373b0d5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a8f5cdb58d74eb6bc748f03373b0d5c" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 46847 } } }
I20260620 14:55:23.089764 27438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:23.089771 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:23.087583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38737 (local address 127.5.118.190:46847)
0620 14:55:23.087785 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:23.087795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:23.087831 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:55:23.087966 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:23.087974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:23.088010 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:23.088266 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:23.089093 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:23.089098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:23.089098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:23.089103 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:23.089150 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:23.089166 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:23.089237 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:23.089412 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:55:23.089456 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260620 14:55:23.090773 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:23.092633 27443 catalog_manager.cc:1347] Generated new cluster ID: 927d1cde860d45ac89d002305c0b9437
I20260620 14:55:23.092684 27443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:23.093252 27436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:23.092061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47263 (local address 127.5.118.190:46847)
0620 14:55:23.092094 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:23.092096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:23.092219 (+ 123us) server_negotiation.cc:407] Connection header received
0620 14:55:23.092302 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:23.092303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:23.092308 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:23.092362 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:23.092997 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:23.092998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:23.092998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:23.093002 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:23.093017 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:23.093021 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:23.093052 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:23.093179 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:55:23.093193 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 14:55:23.106053 27443 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:23.106267 27443 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:23.115545 27443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a8f5cdb58d74eb6bc748f03373b0d5c: Generated new TSK 0
I20260620 14:55:23.115796 27443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:23.124517 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 27364
2026-06-20T14:55:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:23 dist-test-slave-2m5f krb5kdc[27463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:23 dist-test-slave-2m5f krb5kdc[27463](info): set up 2 sockets
Jun 20 14:55:23 dist-test-slave-2m5f krb5kdc[27463](info): commencing operation
krb5kdc: starting...
W20260620 14:55:24.224433 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:24 dist-test-slave-2m5f krb5kdc[27463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:55:24.351599 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:41387
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:44255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:41387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:24.435062 27476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:24.435295 27476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:24.435320 27476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:24.436575 27476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:24.436622 27476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:24.436636 27476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:24.436650 27476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:24.436662 27476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:24.438194 27476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:44255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:41387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:41387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:24.438426 27476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:24.438618 27476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:24.443704 27481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:24.443874 27482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:24.443902 27484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:24.444324 27476 server_base.cc:1034] running on GCE node
I20260620 14:55:24.444602 27476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:24.444936 27476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:24.446131 27476 hybrid_clock.cc:648] HybridClock initialized: now 1781967324446108 us; error 43 us; skew 500 ppm
Jun 20 14:55:24 dist-test-slave-2m5f krb5kdc[27463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967324, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:24.448887 27476 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:55:24.449365 27476 webserver.cc:460] Webserver started at http://127.5.118.190:43101/ using document root <none> and password file <none>
I20260620 14:55:24.449558 27476 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:24.449599 27476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:24.449694 27476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:24.450743 27476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "8fa9e741cb9d416795baddbab0419eb3"
format_stamp: "Formatted at 2026-06-20 14:55:24 on dist-test-slave-2m5f"
server_key: "1054e1d378781e126919c795256ba659"
server_key_iv: "b0f97b0bd8cec6c223b519fe5065a365"
server_key_version: "encryptionkey@0"
I20260620 14:55:24.451061 27476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "8fa9e741cb9d416795baddbab0419eb3"
format_stamp: "Formatted at 2026-06-20 14:55:24 on dist-test-slave-2m5f"
server_key: "1054e1d378781e126919c795256ba659"
server_key_iv: "b0f97b0bd8cec6c223b519fe5065a365"
server_key_version: "encryptionkey@0"
I20260620 14:55:24.452479 27476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:55:24.453253 27491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:24.453467 27476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:24.453567 27476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "8fa9e741cb9d416795baddbab0419eb3"
format_stamp: "Formatted at 2026-06-20 14:55:24 on dist-test-slave-2m5f"
server_key: "1054e1d378781e126919c795256ba659"
server_key_iv: "b0f97b0bd8cec6c223b519fe5065a365"
server_key_version: "encryptionkey@0"
I20260620 14:55:24.453647 27476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:24.474351 27476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:24.476853 27476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:24.477010 27476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:24.481091 27476 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:41387
I20260620 14:55:24.481186 27543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:41387 every 8 connection(s)
I20260620 14:55:24.481523 27476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:24.482470 27544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:24.484898 27544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3: Bootstrap starting.
I20260620 14:55:24.485442 27544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:24.485700 27544 log.cc:824] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:24.486471 27544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3: No bootstrap required, opened a new log
I20260620 14:55:24.486749 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 27476
I20260620 14:55:24.486850 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:24.487030 5594 external_mini_cluster.cc:1428] Setting key 3a7ecbf9525234384333edbf0f418c73
I20260620 14:55:24.489217 27544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9e741cb9d416795baddbab0419eb3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41387 } }
I20260620 14:55:24.489462 27544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:24.489504 27544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa9e741cb9d416795baddbab0419eb3, State: Initialized, Role: FOLLOWER
I20260620 14:55:24.489652 27544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9e741cb9d416795baddbab0419eb3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41387 } }
I20260620 14:55:24.489746 27544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:24.489785 27544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:24.489842 27544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:24.490499 27544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9e741cb9d416795baddbab0419eb3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41387 } }
I20260620 14:55:24.490589 27544 leader_election.cc:302] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fa9e741cb9d416795baddbab0419eb3; no voters:
I20260620 14:55:24.490824 27544 leader_election.cc:288] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:24.490860 27549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:24.490978 27549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 8fa9e741cb9d416795baddbab0419eb3, State: Running, Role: LEADER
I20260620 14:55:24.491142 27549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9e741cb9d416795baddbab0419eb3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41387 } }
I20260620 14:55:24.491207 27544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:24.491535 27549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fa9e741cb9d416795baddbab0419eb3. Latest consensus state: current_term: 1 leader_uuid: "8fa9e741cb9d416795baddbab0419eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9e741cb9d416795baddbab0419eb3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41387 } } }
I20260620 14:55:24.491671 27549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:24.492198 27556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:24.492431 27550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fa9e741cb9d416795baddbab0419eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fa9e741cb9d416795baddbab0419eb3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41387 } } }
I20260620 14:55:24.492576 27550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:24.492782 27556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:24.494525 27556 catalog_manager.cc:1347] Generated new cluster ID: cdd2c7e40d394b1a8e913d17310a15f1
I20260620 14:55:24.494578 27556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 14:55:24 dist-test-slave-2m5f krb5kdc[27463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:24.499382 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:24.487629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60611 (local address 127.5.118.190:41387)
0620 14:55:24.487805 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:24.487813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:24.487839 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:55:24.489083 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:24.489093 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:24.489158 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:24.489284 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:24.490151 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:24.491317 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:24.492485 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:24.493151 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:24.493276 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:24.495969 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:24.495973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:24.495973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:24.495986 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:24.497779 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:24.498179 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:24.498182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:24.498182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:24.498288 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:24.498503 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:24.498504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:24.498504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:24.498748 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:24.498924 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:24.499020 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:55:24.499165 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260620 14:55:24.500430 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:24.507115 27556 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:24.507831 27556 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:24.508209 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:24.501802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43517 (local address 127.5.118.190:41387)
0620 14:55:24.501841 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:24.501843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:24.501977 (+ 134us) server_negotiation.cc:407] Connection header received
0620 14:55:24.502045 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:24.502046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:24.502056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:24.502106 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:24.502610 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:24.503109 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:24.503816 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:24.504033 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:24.504109 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:24.505376 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:24.505377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:24.505378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:24.505384 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:24.506896 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:24.507375 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:24.507377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:24.507377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:24.507432 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:24.507642 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:24.507642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:24.507643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:24.507757 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:24.507862 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:24.507994 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:55:24.508082 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:55:24.514424 27556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fa9e741cb9d416795baddbab0419eb3: Generated new TSK 0
I20260620 14:55:24.514633 27556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:24.523085 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 27476
2026-06-20T14:55:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:24 dist-test-slave-2m5f krb5kdc[27575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:24 dist-test-slave-2m5f krb5kdc[27575](info): set up 2 sockets
Jun 20 14:55:24 dist-test-slave-2m5f krb5kdc[27575](info): commencing operation
krb5kdc: starting...
W20260620 14:55:25.564306 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:25 dist-test-slave-2m5f krb5kdc[27575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:55:25.688395 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:41447
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:44445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:41447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:25.773578 27588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:25.773809 27588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:25.773833 27588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:25.775041 27588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:25.775086 27588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:25.775100 27588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:25.775113 27588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:25.775166 27588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:25.776818 27588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:44445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:41447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:41447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:25.777094 27588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:25.777338 27588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:55:25.782506 27588 server_base.cc:1034] running on GCE node
W20260620 14:55:25.782464 27593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:25.782472 27596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:25.782472 27594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:25.783100 27588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:25.783435 27588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:25.784636 27588 hybrid_clock.cc:648] HybridClock initialized: now 1781967325784626 us; error 41 us; skew 500 ppm
Jun 20 14:55:25 dist-test-slave-2m5f krb5kdc[27575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967325, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:25.787158 27588 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:55:25.787554 27588 webserver.cc:460] Webserver started at http://127.5.118.190:45045/ using document root <none> and password file <none>
I20260620 14:55:25.787751 27588 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:25.787792 27588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:25.787879 27588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:25.788959 27588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "5fb31f6342474366b492d2ceda9df61f"
format_stamp: "Formatted at 2026-06-20 14:55:25 on dist-test-slave-2m5f"
server_key: "151695e80f1e11d5a468af8558530bcf"
server_key_iv: "1f103b45094bad7e02f533b7d0ba8975"
server_key_version: "encryptionkey@0"
I20260620 14:55:25.789337 27588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "5fb31f6342474366b492d2ceda9df61f"
format_stamp: "Formatted at 2026-06-20 14:55:25 on dist-test-slave-2m5f"
server_key: "151695e80f1e11d5a468af8558530bcf"
server_key_iv: "1f103b45094bad7e02f533b7d0ba8975"
server_key_version: "encryptionkey@0"
I20260620 14:55:25.790987 27588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 14:55:25.791850 27603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:25.792165 27588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 14:55:25.792316 27588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "5fb31f6342474366b492d2ceda9df61f"
format_stamp: "Formatted at 2026-06-20 14:55:25 on dist-test-slave-2m5f"
server_key: "151695e80f1e11d5a468af8558530bcf"
server_key_iv: "1f103b45094bad7e02f533b7d0ba8975"
server_key_version: "encryptionkey@0"
I20260620 14:55:25.792395 27588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:25.806967 27588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:25.809270 27588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:25.809435 27588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:25.813476 27588 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:41447
I20260620 14:55:25.813558 27655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:41447 every 8 connection(s)
I20260620 14:55:25.813905 27588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:25.814858 27656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:25.817157 27656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f: Bootstrap starting.
I20260620 14:55:25.817716 27656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:25.817960 27656 log.cc:824] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:25.818699 27656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f: No bootstrap required, opened a new log
I20260620 14:55:25.820392 27656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb31f6342474366b492d2ceda9df61f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41447 } }
I20260620 14:55:25.820539 27656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:25.820562 27656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb31f6342474366b492d2ceda9df61f, State: Initialized, Role: FOLLOWER
I20260620 14:55:25.820678 27656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb31f6342474366b492d2ceda9df61f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41447 } }
I20260620 14:55:25.820765 27656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:25.820802 27656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:25.820878 27656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:25.821785 27656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb31f6342474366b492d2ceda9df61f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41447 } }
I20260620 14:55:25.821916 27656 leader_election.cc:302] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fb31f6342474366b492d2ceda9df61f; no voters:
I20260620 14:55:25.822104 27656 leader_election.cc:288] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:25.822248 27659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:25.822434 27659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [term 1 LEADER]: Becoming Leader. State: Replica: 5fb31f6342474366b492d2ceda9df61f, State: Running, Role: LEADER
I20260620 14:55:25.822440 27656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:25.822592 27659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb31f6342474366b492d2ceda9df61f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41447 } }
I20260620 14:55:25.822997 27661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fb31f6342474366b492d2ceda9df61f. Latest consensus state: current_term: 1 leader_uuid: "5fb31f6342474366b492d2ceda9df61f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb31f6342474366b492d2ceda9df61f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41447 } } }
I20260620 14:55:25.822991 27660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fb31f6342474366b492d2ceda9df61f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fb31f6342474366b492d2ceda9df61f" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 41447 } } }
I20260620 14:55:25.823102 27661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:25.823102 27660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:25.823628 27668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:25.823709 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 27588
I20260620 14:55:25.823837 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:25.824071 5594 external_mini_cluster.cc:1428] Setting key 3f3cbfc225343bff8e4285af727921e5
I20260620 14:55:25.824196 27668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:25.825899 27668 catalog_manager.cc:1347] Generated new cluster ID: 56d808ba3395481793bdb86eef5275af
I20260620 14:55:25.825953 27668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:25.829697 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:25.824770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33941 (local address 127.5.118.190:41447)
0620 14:55:25.825073 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:25.825084 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:25.825131 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:55:25.825259 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:25.825267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:25.825325 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:25.825476 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:25.825700 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:25.826724 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:25.827544 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:25.827978 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:25.828096 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:25.829014 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:25.829018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:25.829018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:25.829205 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:25.829236 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:25.829247 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:25.829325 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:25.829403 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:55:25.829470 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":81,"threads_started":1}
I20260620 14:55:25.830651 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:25.835219 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:25.832011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36037 (local address 127.5.118.190:41447)
0620 14:55:25.832051 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:25.832053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:25.832477 (+ 424us) server_negotiation.cc:407] Connection header received
0620 14:55:25.832548 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:25.832549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:25.832563 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:25.832617 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:25.832746 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:25.833237 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:25.833959 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:25.834170 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:25.834256 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:25.834960 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:25.834961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:25.834961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:25.834967 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:25.834982 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:25.834986 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:25.835041 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:25.835099 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:55:25.835123 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:55:25.851568 27668 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:25.852474 27668 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:25.857327 27668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fb31f6342474366b492d2ceda9df61f: Generated new TSK 0
I20260620 14:55:25.857564 27668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:25.859818 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 27588
2026-06-20T14:55:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:25 dist-test-slave-2m5f krb5kdc[27687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:25 dist-test-slave-2m5f krb5kdc[27687](info): set up 2 sockets
Jun 20 14:55:25 dist-test-slave-2m5f krb5kdc[27687](info): commencing operation
krb5kdc: starting...
W20260620 14:55:26.892704 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:26 dist-test-slave-2m5f krb5kdc[27687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:55:27.014832 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:44497
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:44497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:27.096861 27700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:27.097070 27700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:27.097093 27700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:27.098266 27700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:27.098312 27700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:27.098326 27700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:27.098340 27700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:27.098351 27700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:27.099973 27700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:44497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:44497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:27.100268 27700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:27.100508 27700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:27.105530 27705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:27.105517 27706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:27.105517 27708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:27.105671 27700 server_base.cc:1034] running on GCE node
I20260620 14:55:27.106113 27700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:27.106412 27700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:27.107589 27700 hybrid_clock.cc:648] HybridClock initialized: now 1781967327107581 us; error 45 us; skew 500 ppm
Jun 20 14:55:27 dist-test-slave-2m5f krb5kdc[27687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967327, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:27.110431 27700 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:55:27.110954 27700 webserver.cc:460] Webserver started at http://127.5.118.190:35341/ using document root <none> and password file <none>
I20260620 14:55:27.111227 27700 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:27.111302 27700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:27.111446 27700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:27.112663 27700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "7010ca9d41354fc2b6f2b1299e127676"
format_stamp: "Formatted at 2026-06-20 14:55:27 on dist-test-slave-2m5f"
server_key: "74bf9adb7dbc22a9923083eaf33cbe8d"
server_key_iv: "d17845109d8992dcd8a1d6ee2d7cd1d4"
server_key_version: "encryptionkey@0"
I20260620 14:55:27.113027 27700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "7010ca9d41354fc2b6f2b1299e127676"
format_stamp: "Formatted at 2026-06-20 14:55:27 on dist-test-slave-2m5f"
server_key: "74bf9adb7dbc22a9923083eaf33cbe8d"
server_key_iv: "d17845109d8992dcd8a1d6ee2d7cd1d4"
server_key_version: "encryptionkey@0"
I20260620 14:55:27.114481 27700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:55:27.115343 27715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:27.115543 27700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:27.115693 27700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "7010ca9d41354fc2b6f2b1299e127676"
format_stamp: "Formatted at 2026-06-20 14:55:27 on dist-test-slave-2m5f"
server_key: "74bf9adb7dbc22a9923083eaf33cbe8d"
server_key_iv: "d17845109d8992dcd8a1d6ee2d7cd1d4"
server_key_version: "encryptionkey@0"
I20260620 14:55:27.115782 27700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:27.119251 27700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:27.121487 27700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:27.121649 27700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:27.125790 27700 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:44497
I20260620 14:55:27.125927 27767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:44497 every 8 connection(s)
I20260620 14:55:27.126194 27700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:27.127040 27768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:27.129382 27768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676: Bootstrap starting.
I20260620 14:55:27.129698 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 27700
I20260620 14:55:27.129827 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:27.130045 27768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:27.130077 5594 external_mini_cluster.cc:1428] Setting key 5e95b0f157960883b81aa9c0d91694a7
I20260620 14:55:27.130404 27768 log.cc:824] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:27.131177 27768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676: No bootstrap required, opened a new log
I20260620 14:55:27.133013 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:27.130671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58299 (local address 127.5.118.190:44497)
0620 14:55:27.130874 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:27.130883 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:27.130970 (+ 87us) server_negotiation.cc:407] Connection header received
0620 14:55:27.131587 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:27.131598 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:27.131633 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:27.131791 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:27.132479 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:27.132482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:27.132483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:27.132513 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:27.132561 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:27.132576 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:27.132674 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:27.132753 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 14:55:27.132796 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260620 14:55:27.134090 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:27.134114 27768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7010ca9d41354fc2b6f2b1299e127676" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44497 } }
I20260620 14:55:27.134284 27768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:55:27.134315 27768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7010ca9d41354fc2b6f2b1299e127676, State: Initialized, Role: FOLLOWER
I20260620 14:55:27.134430 27768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7010ca9d41354fc2b6f2b1299e127676" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44497 } }
I20260620 14:55:27.134506 27768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:55:27.134538 27768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:55:27.134591 27768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:27.135397 27768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7010ca9d41354fc2b6f2b1299e127676" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44497 } }
I20260620 14:55:27.135514 27768 leader_election.cc:302] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7010ca9d41354fc2b6f2b1299e127676; no voters:
I20260620 14:55:27.135685 27768 leader_election.cc:288] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:55:27.135939 27779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:27.136032 27768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:27.136107 27779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [term 1 LEADER]: Becoming Leader. State: Replica: 7010ca9d41354fc2b6f2b1299e127676, State: Running, Role: LEADER
I20260620 14:55:27.136271 27779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7010ca9d41354fc2b6f2b1299e127676" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44497 } }
I20260620 14:55:27.136608 27779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7010ca9d41354fc2b6f2b1299e127676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7010ca9d41354fc2b6f2b1299e127676" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44497 } } }
I20260620 14:55:27.136701 27779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:27.136837 27779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7010ca9d41354fc2b6f2b1299e127676. Latest consensus state: current_term: 1 leader_uuid: "7010ca9d41354fc2b6f2b1299e127676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7010ca9d41354fc2b6f2b1299e127676" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 44497 } } }
I20260620 14:55:27.136910 27787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:27.136996 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:27.135723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53919 (local address 127.5.118.190:44497)
0620 14:55:27.135785 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:27.135787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:27.135854 (+ 67us) server_negotiation.cc:407] Connection header received
0620 14:55:27.135916 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:27.135917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:27.135945 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:27.136001 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:27.136638 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:27.136639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:27.136640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:55:27.136644 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:27.136660 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:55:27.136665 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:27.136726 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:27.136924 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 14:55:27.136937 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:55:27.137065 27779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:27.137622 27787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:27.139220 27787 catalog_manager.cc:1347] Generated new cluster ID: 3b67f8f7291c40c9975460a7c81f32cc
I20260620 14:55:27.139269 27787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:27.147475 27787 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:27.147706 27787 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:27.154223 27787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7010ca9d41354fc2b6f2b1299e127676: Generated new TSK 0
I20260620 14:55:27.154443 27787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:55:27.157105 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 27700
2026-06-20T14:55:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1296 ms)
[----------] 18 tests from AuthTokenIssuingTest (24406 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:27 dist-test-slave-2m5f krb5kdc[27799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:27 dist-test-slave-2m5f krb5kdc[27799](info): set up 2 sockets
Jun 20 14:55:27 dist-test-slave-2m5f krb5kdc[27799](info): commencing operation
krb5kdc: starting...
W20260620 14:55:28.240523 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:55:28.377044 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:45519
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:45519,127.5.118.189:36807,127.5.118.188:33941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:28.460843 27812 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:55:28.461139 27812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:28.461172 27812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:28.461212 27812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:28.462394 27812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:28.462483 27812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:28.462507 27812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:28.462527 27812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:28.462556 27812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:28.464120 27812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:45519,127.5.118.189:36807,127.5.118.188:33941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:28.464452 27812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:28.464702 27812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:28.470120 27820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:28.470124 27818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:28.470118 27817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:28.470474 27812 server_base.cc:1034] running on GCE node
I20260620 14:55:28.470809 27812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:28.471225 27812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:28.472379 27812 hybrid_clock.cc:648] HybridClock initialized: now 1781967328472349 us; error 44 us; skew 500 ppm
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:28.478756 27812 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:55:28.479214 27812 webserver.cc:460] Webserver started at http://127.5.118.190:42405/ using document root <none> and password file <none>
I20260620 14:55:28.479473 27812 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:28.479553 27812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:28.479719 27812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:28.480917 27812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "18e3d719e40b4503815fddd412fac961"
format_stamp: "Formatted at 2026-06-20 14:55:28 on dist-test-slave-2m5f"
server_key: "60454aaf4112f9c25fbb8fb97c23687f"
server_key_iv: "802b37da634b2f7dae8a49cc5d3ff247"
server_key_version: "encryptionkey@0"
I20260620 14:55:28.481302 27812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "18e3d719e40b4503815fddd412fac961"
format_stamp: "Formatted at 2026-06-20 14:55:28 on dist-test-slave-2m5f"
server_key: "60454aaf4112f9c25fbb8fb97c23687f"
server_key_iv: "802b37da634b2f7dae8a49cc5d3ff247"
server_key_version: "encryptionkey@0"
I20260620 14:55:28.482817 27812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:55:28.483791 27827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:28.484118 27812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:28.484362 27812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "18e3d719e40b4503815fddd412fac961"
format_stamp: "Formatted at 2026-06-20 14:55:28 on dist-test-slave-2m5f"
server_key: "60454aaf4112f9c25fbb8fb97c23687f"
server_key_iv: "802b37da634b2f7dae8a49cc5d3ff247"
server_key_version: "encryptionkey@0"
I20260620 14:55:28.484462 27812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:28.503743 27812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:28.506349 27812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:28.506538 27812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:28.510840 27812 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:45519
I20260620 14:55:28.510953 27879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:45519 every 8 connection(s)
I20260620 14:55:28.511304 27812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:28.512470 27880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:28.512607 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 27812
I20260620 14:55:28.512717 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:28.512938 5594 external_mini_cluster.cc:1428] Setting key 4a6f60856b38d3e87591a59356094255
I20260620 14:55:28.514962 27880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.118.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.189@KRBTEST.COM" created.
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:28.529393 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.515844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.5.118.190:45519)
0620 14:55:28.516036 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.516042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.516106 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:55:28.516826 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.516829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.516851 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.516968 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.517540 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.518381 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.519326 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.519634 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.519777 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.525422 (+ 5645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.525434 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.525435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.525445 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.527335 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.527900 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.527904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.527904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.528002 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.528329 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.528330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.528331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.528594 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.528708 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.528938 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.529097 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":105,"threads_started":1}
I20260620 14:55:28.529392 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.515748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:45519 (local address 127.0.0.1:51942)
0620 14:55:28.515948 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.515997 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:28.516635 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:28.516977 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:28.516979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:28.517275 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:28.517429 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.517433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.518504 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.518505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.519197 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.519200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.519760 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.519761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.519823 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.520477 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:28.520487 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:28.525301 (+ 4814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:28.527454 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.527462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.527464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.527734 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.528087 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.528088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.528088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.528244 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.528682 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:28.528688 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:28.528771 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 14:55:28.528887 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 14:55:28.529168 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":111,"thread_start_us":90,"threads_started":1}
I20260620 14:55:28.530484 27880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:55:28.530951 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.530784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:36807 (local address 127.0.0.1:56808)
0620 14:55:28.530852 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.530902 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:55:28.531147 27829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.118.189:36807: connect: Connection refused (error 111)
I20260620 14:55:28.531685 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.531587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:36807 (local address 127.0.0.1:56812)
0620 14:55:28.531639 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.531666 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:55:28.531952 27880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.189:36807: Network error: Client connection negotiation failed: client connection to 127.5.118.189:36807: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab.
Entry for principal kudu/127.5.118.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab.
I20260620 14:55:28.543586 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.189:36807
--webserver_interface=127.5.118.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab
--principal=kudu/127.5.118.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:45519,127.5.118.189:36807,127.5.118.188:33941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:55:28.581247 27880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } attempt: 1
I20260620 14:55:28.581861 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.581712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:36807 (local address 127.0.0.1:56814)
0620 14:55:28.581787 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.581827 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:55:28.582288 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.582147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:36807 (local address 127.0.0.1:56828)
0620 14:55:28.582213 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.582258 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 14:55:28.582477 27880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.189:36807: Network error: Client connection negotiation failed: client connection to 127.5.118.189:36807: connect: Connection refused (error 111)
W20260620 14:55:28.627702 27887 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:55:28.627929 27887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:28.627947 27887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:28.627969 27887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:28.629341 27887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:28.629391 27887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:28.629407 27887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:28.629420 27887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:28.629432 27887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:28.630995 27887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:45519,127.5.118.189:36807,127.5.118.188:33941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab
--principal=kudu/127.5.118.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.189:36807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.118.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:28.631289 27887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:28.631546 27887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:28.636545 27892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:28.636557 27893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:28.636557 27895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:28.636879 27887 server_base.cc:1034] running on GCE node
I20260620 14:55:28.637132 27887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:28.637446 27887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:28.638603 27887 hybrid_clock.cc:648] HybridClock initialized: now 1781967328638581 us; error 44 us; skew 500 ppm
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:28.641518 27887 init.cc:377] Logged in from keytab as kudu/127.5.118.189@KRBTEST.COM (short username kudu)
I20260620 14:55:28.642077 27887 webserver.cc:460] Webserver started at http://127.5.118.189:35861/ using document root <none> and password file <none>
I20260620 14:55:28.642412 27887 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:28.642493 27887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:28.642648 27887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:28.643728 27887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/data/instance:
uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18"
format_stamp: "Formatted at 2026-06-20 14:55:28 on dist-test-slave-2m5f"
server_key: "24ac04d0bc9b256017ccbfa543f47479"
server_key_iv: "4e271971a7b1e2cd4e6b8e376cbc53d2"
server_key_version: "encryptionkey@0"
I20260620 14:55:28.644132 27887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/wal/instance:
uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18"
format_stamp: "Formatted at 2026-06-20 14:55:28 on dist-test-slave-2m5f"
server_key: "24ac04d0bc9b256017ccbfa543f47479"
server_key_iv: "4e271971a7b1e2cd4e6b8e376cbc53d2"
server_key_version: "encryptionkey@0"
I20260620 14:55:28.645591 27887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:28.646443 27902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:28.646662 27887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:28.646759 27887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/wal
uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18"
format_stamp: "Formatted at 2026-06-20 14:55:28 on dist-test-slave-2m5f"
server_key: "24ac04d0bc9b256017ccbfa543f47479"
server_key_iv: "4e271971a7b1e2cd4e6b8e376cbc53d2"
server_key_version: "encryptionkey@0"
I20260620 14:55:28.646816 27887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:28.650682 27880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } attempt: 2
I20260620 14:55:28.651140 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.650998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:36807 (local address 127.0.0.1:56834)
0620 14:55:28.651072 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.651111 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:55:28.651504 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.651419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:36807 (local address 127.0.0.1:56848)
0620 14:55:28.651460 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.651486 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:36807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:55:28.651645 27880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.189:36807: Network error: Client connection negotiation failed: client connection to 127.5.118.189:36807: connect: Connection refused (error 111)
I20260620 14:55:28.659361 27887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:28.661937 27887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:28.662096 27887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:28.666330 27887 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.189:36807
I20260620 14:55:28.666399 27954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.189:36807 every 8 connection(s)
I20260620 14:55:28.666751 27887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
I20260620 14:55:28.667778 27955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:28.669490 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 27887
I20260620 14:55:28.669575 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-1/wal/instance
I20260620 14:55:28.669479 27955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:55:28.669834 5594 external_mini_cluster.cc:1428] Setting key 0e862efa96b10f4a3de6958f69de5e53
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.118.188@KRBTEST.COM; defaulting to no policy
I20260620 14:55:28.681033 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.670415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.5.118.190:45519)
0620 14:55:28.670482 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.670485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.670670 (+ 185us) server_negotiation.cc:407] Connection header received
0620 14:55:28.671336 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.671337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.671351 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.671409 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.672337 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.672826 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.674493 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.674709 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.674776 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.677596 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.677598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.677599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.677607 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.679207 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.679631 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.679632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.679632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.679720 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.680082 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.680083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.680084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.680404 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.680473 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.680881 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.680963 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:55:28.681245 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.670358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:45519 (local address 127.0.0.1:51946)
0620 14:55:28.670544 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.670581 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:28.671154 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:28.671518 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:28.671522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:28.671901 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:28.672243 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.672250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.672931 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.672932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.674379 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.674382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.674828 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.674830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.675017 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.675676 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:28.675688 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:28.677469 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:28.679318 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.679325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.679328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.679560 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.679795 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.679796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.679796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.679902 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.680495 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:28.680501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:28.680673 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 14:55:28.680804 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:55:28.680948 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.188@KRBTEST.COM" created.
I20260620 14:55:28.681981 27955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
I20260620 14:55:28.691015 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.682319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:36807 (local address 127.0.0.1:56852)
0620 14:55:28.682381 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.682392 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:28.682436 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:28.682671 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:28.682673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:28.682852 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:28.682921 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.682923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.683825 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.683826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.684691 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.684694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.685291 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.685292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.685377 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.685907 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:28.685917 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:28.687232 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:28.689650 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.689651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.689652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.689853 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.690134 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.690135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.690135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.690210 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.690625 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:28.690626 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:28.690700 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 14:55:28.690776 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 14:55:28.690869 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:55:28.691025 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.682407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.5.118.189:36807)
0620 14:55:28.682540 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.682543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.682559 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:55:28.682575 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.682577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.682587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.682638 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.683005 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.683722 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.684830 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.685143 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.685289 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.687363 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.687372 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.687373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.687381 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.689551 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.689961 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.689966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.689966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.690031 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.690300 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.690301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.690301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.690514 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.690613 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.690849 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.690928 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260620 14:55:28.692027 27955 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:55:28.692538 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.692367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55640)
0620 14:55:28.692450 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.692514 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 14:55:28.692729 27904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
I20260620 14:55:28.693389 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.693122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55650)
0620 14:55:28.693267 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.693303 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 14:55:28.693559 27955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:33941: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab.
Entry for principal kudu/127.5.118.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab.
I20260620 14:55:28.699689 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.188:33941
--webserver_interface=127.5.118.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab
--principal=kudu/127.5.118.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:34369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:45519,127.5.118.189:36807,127.5.118.188:33941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:55:28.742870 27880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } attempt: 3
I20260620 14:55:28.742875 27955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } attempt: 1
I20260620 14:55:28.743361 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.743217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55660)
0620 14:55:28.743285 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.743323 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:55:28.743839 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.743718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55664)
0620 14:55:28.743780 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.743808 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:55:28.743934 27955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:33941: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
I20260620 14:55:28.751304 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.743250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:36807 (local address 127.0.0.1:56858)
0620 14:55:28.743296 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.743306 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:28.743350 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:28.743631 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:28.743632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:28.743836 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:28.743931 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.743934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.744983 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.744984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.745547 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.745550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.746143 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.746145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.746250 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.746751 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:28.746757 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:28.747988 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:28.749947 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.749948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.749948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.750331 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.750621 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.750622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.750623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.750707 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.750974 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:28.750976 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:28.751029 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 14:55:28.751100 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 14:55:28.751202 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:55:28.751335 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.743343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.5.118.189:36807)
0620 14:55:28.743387 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.743389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.743399 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:55:28.743428 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.743429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.743439 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.744023 (+ 584us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.744068 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.744880 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.745663 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.746017 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.746110 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.748078 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.748079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.748079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.748087 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.749853 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.750452 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.750453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.750454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.750518 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.750780 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.750780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.750780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.750906 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.750960 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.751207 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.751274 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:55:28.752321 27880 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:55:28.752753 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.752621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55678)
0620 14:55:28.752685 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.752724 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:55:28.753149 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.753033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55686)
0620 14:55:28.753083 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.753129 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:55:28.753311 27880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:33941: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
I20260620 14:55:28.784477 27880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } attempt: 1
I20260620 14:55:28.785040 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.784896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55688)
0620 14:55:28.784969 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.785013 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 14:55:28.785494 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.785400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55698)
0620 14:55:28.785447 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.785474 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 14:55:28.785660 27880 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:33941: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
W20260620 14:55:28.789337 27962 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:55:28.789561 27962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:28.789578 27962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:28.789601 27962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:28.790768 27962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:28.790814 27962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:28.790829 27962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:28.790843 27962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:28.790855 27962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:28.792479 27962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:34369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:45519,127.5.118.189:36807,127.5.118.188:33941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab
--principal=kudu/127.5.118.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.188:33941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.118.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:28.792690 27962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:28.792877 27962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:28.797917 27967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:28.797927 27968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:28.797986 27970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:28.798309 27962 server_base.cc:1034] running on GCE node
I20260620 14:55:28.798643 27962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:28.798950 27962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:28.800127 27962 hybrid_clock.cc:648] HybridClock initialized: now 1781967328800109 us; error 53 us; skew 500 ppm
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:28.802681 27962 init.cc:377] Logged in from keytab as kudu/127.5.118.188@KRBTEST.COM (short username kudu)
I20260620 14:55:28.803133 27962 webserver.cc:460] Webserver started at http://127.5.118.188:36259/ using document root <none> and password file <none>
I20260620 14:55:28.803395 27962 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:28.803471 27962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:28.803624 27962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:28.804853 27962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/data/instance:
uuid: "69ba357d3b63493eb956d5d63e0ee60e"
format_stamp: "Formatted at 2026-06-20 14:55:28 on dist-test-slave-2m5f"
server_key: "5d32c83575c6dbd432f1a9ba828f41c4"
server_key_iv: "c4757612bb40dc6a474aa01f20bac5ed"
server_key_version: "encryptionkey@0"
I20260620 14:55:28.805246 27962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/wal/instance:
uuid: "69ba357d3b63493eb956d5d63e0ee60e"
format_stamp: "Formatted at 2026-06-20 14:55:28 on dist-test-slave-2m5f"
server_key: "5d32c83575c6dbd432f1a9ba828f41c4"
server_key_iv: "c4757612bb40dc6a474aa01f20bac5ed"
server_key_version: "encryptionkey@0"
I20260620 14:55:28.806607 27962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:55:28.807343 27977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:28.807520 27962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 14:55:28.807618 27962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/wal
uuid: "69ba357d3b63493eb956d5d63e0ee60e"
format_stamp: "Formatted at 2026-06-20 14:55:28 on dist-test-slave-2m5f"
server_key: "5d32c83575c6dbd432f1a9ba828f41c4"
server_key_iv: "c4757612bb40dc6a474aa01f20bac5ed"
server_key_version: "encryptionkey@0"
I20260620 14:55:28.807703 27962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:28.812079 27955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } attempt: 2
I20260620 14:55:28.812557 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.812423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55700)
0620 14:55:28.812483 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.812527 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 14:55:28.812896 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.812816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55716)
0620 14:55:28.812857 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.812879 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 14:55:28.813048 27955 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:33941: Network error: Client connection negotiation failed: client connection to 127.5.118.188:33941: connect: Connection refused (error 111)
I20260620 14:55:28.830487 27962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:28.832793 27962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:28.832959 27962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:28.837175 27962 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.188:33941
I20260620 14:55:28.837391 28029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.188:33941 every 8 connection(s)
I20260620 14:55:28.837643 27962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
I20260620 14:55:28.838780 28030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:28.840585 28030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:55:28.845314 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 27962
I20260620 14:55:28.845448 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967053866185-5594-0/minicluster-data/master-2/wal/instance
I20260620 14:55:28.845700 5594 external_mini_cluster.cc:1428] Setting key 7718e21f5fecf1fe18db8390a8a56bee
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:28.850937 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.841388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.5.118.190:45519)
0620 14:55:28.841468 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.841470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.841607 (+ 137us) server_negotiation.cc:407] Connection header received
0620 14:55:28.842097 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.842099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.842108 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.842170 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.842917 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.843396 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.844595 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.844793 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.844859 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.847417 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.847419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.847419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.847427 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.848919 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.849685 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.849687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.849688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.849763 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.850093 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.850094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.850095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.850255 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.850364 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.850732 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.850833 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:55:28.851145 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.841290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:45519 (local address 127.0.0.1:51954)
0620 14:55:28.841468 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.841514 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:28.842004 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:28.842247 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:28.842250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:28.842612 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:28.842818 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.842824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.843488 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.843489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.844502 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.844504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.844888 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.844889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.844999 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.845754 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:28.845765 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:28.847302 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:28.849032 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.849040 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.849043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.849304 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.849854 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.849855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.849856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.850002 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.850354 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:28.850362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:28.850529 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 14:55:28.850664 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 14:55:28.850848 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
I20260620 14:55:28.852202 28030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:55:28.855415 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.846153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.5.118.188:33941)
0620 14:55:28.846446 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.846450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.846468 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:55:28.846486 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.846488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.846504 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.846589 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.847396 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.848074 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.848905 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.849357 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.849457 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.852220 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.852227 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.852228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.852235 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.854077 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.854485 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.854489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.854489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.854554 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.854783 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.854784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.854784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.854944 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.855125 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.855200 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.855289 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":85,"threads_started":1}
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
I20260620 14:55:28.860585 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.852491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:36807 (local address 127.0.0.1:56868)
0620 14:55:28.852857 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.852867 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:28.852950 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:28.853180 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:28.853181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:28.853454 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:28.853505 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.853508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.854341 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.854343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.854911 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.854912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.855403 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.855404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.855477 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.855920 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:28.855925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:28.857597 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:28.859503 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.859504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.859505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.859692 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.859922 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.859923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.859923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.859991 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.860329 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:28.860330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:28.860372 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:55:28.860435 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 14:55:28.860516 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349}
I20260620 14:55:28.860667 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.852570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.5.118.189:36807)
0620 14:55:28.852630 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.852633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.852935 (+ 302us) server_negotiation.cc:407] Connection header received
0620 14:55:28.853020 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.853021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.853034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.853136 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.853602 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.854124 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.855032 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.855289 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.855403 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.857729 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.857732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.857732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.857740 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.859394 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.859775 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.859776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.859777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.859839 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.860073 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.860074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.860074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.860254 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.860327 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.860494 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.860573 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:28.860848 27880 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } attempt: 2
I20260620 14:55:28.861060 28030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:55:28.864976 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.856481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51962 (local address 127.5.118.190:45519)
0620 14:55:28.856537 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.856540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.856576 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:55:28.856646 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.856647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.856656 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.856713 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.857355 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.857872 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.858580 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.858798 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.858890 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.861409 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.861411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.861411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.861420 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.863064 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.863456 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.863457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.863457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.863512 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.863751 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.863753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.863753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.863898 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.864077 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.864675 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.864756 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
I20260620 14:55:28.869354 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.861137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55742)
0620 14:55:28.861223 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.861234 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:28.861271 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:28.861636 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:28.861637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:28.861810 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:28.861866 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.861868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.862831 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.862832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.863352 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.863354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.863898 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.863900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.863965 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.864392 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:28.864396 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:28.865750 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:28.867994 (+ 2244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.867995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.867996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.868242 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.868594 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.868595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.868595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.868660 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.868994 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:28.868995 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:28.869045 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 14:55:28.869131 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 14:55:28.869239 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 14:55:28.869379 28034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.861346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.5.118.188:33941)
0620 14:55:28.861461 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.861463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.861495 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:55:28.861515 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.861515 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.861537 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.861951 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.861975 (+ 24us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.862722 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.863464 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.863795 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.864134 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.865831 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.865832 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.865832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.865840 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.867897 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.868307 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.868308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.868309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.868371 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.868741 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.868743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.868743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.868897 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.868971 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.869176 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.869283 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 14:55:28.870697 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.861271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55744)
0620 14:55:28.861316 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.861326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:28.861357 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:28.861673 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:28.861674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:28.861811 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:28.861854 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.861856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.862829 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.862830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.863621 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.863624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.864058 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.864059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.864117 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.864617 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:28.864623 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:28.866372 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:28.869564 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.869565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.869566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.869763 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.869962 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.869962 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.869963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.870018 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.870311 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:28.870312 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:28.870367 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 14:55:28.870451 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:55:28.870557 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 14:55:28.870783 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.861358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.5.118.188:33941)
0620 14:55:28.861552 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.861554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.861560 (+ 6us) server_negotiation.cc:407] Connection header received
0620 14:55:28.861575 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.861575 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.861583 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.861920 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.861957 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.862731 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.863736 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.863981 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.864301 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.866466 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.866468 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.866468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.866474 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.869491 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.869847 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.869848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.869848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.869902 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.870081 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.870082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.870083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.870227 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.870292 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.870507 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.870610 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":176,"thread_start_us":115,"threads_started":1}
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
I20260620 14:55:28.871371 27880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961: Bootstrap starting.
I20260620 14:55:28.872107 27880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:28.872556 27880 log.cc:824] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:28.873521 27880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961: No bootstrap required, opened a new log
I20260620 14:55:28.874209 28030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e: Bootstrap starting.
I20260620 14:55:28.874981 28030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:28.875108 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.866005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.5.118.189:36807)
0620 14:55:28.866047 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.866049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.866057 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:55:28.866082 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.866083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.866094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.866152 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.866923 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.867520 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.868729 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.868925 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.869019 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.871810 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.871812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.871812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.871820 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.873746 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.874166 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.874167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.874168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.874231 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.874463 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.874464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.874464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.874627 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.874827 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.874925 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.875008 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:55:28.875332 28030 log.cc:824] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:28.875698 27880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } }
I20260620 14:55:28.875890 27880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e3d719e40b4503815fddd412fac961, State: Initialized, Role: FOLLOWER
I20260620 14:55:28.875995 27880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } }
I20260620 14:55:28.876320 28030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e: No bootstrap required, opened a new log
I20260620 14:55:28.876371 27880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:28.876348 28038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } } }
I20260620 14:55:28.876487 28038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:28.876748 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:28.877737 27854 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "18e3d719e40b4503815fddd412fac961"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51962
I20260620 14:55:28.877991 27854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:55:28.878041 27854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [term 0 FOLLOWER]: Advancing to term 1
W20260620 14:55:28.878288 28049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:55:28.878340 28049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:55:28.878495 28030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } }
I20260620 14:55:28.878597 28030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ba357d3b63493eb956d5d63e0ee60e, State: Initialized, Role: FOLLOWER
I20260620 14:55:28.878656 28030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } }
I20260620 14:55:28.878832 27854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } }
I20260620 14:55:28.878878 28051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } } }
I20260620 14:55:28.878943 28030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:28.878952 28051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:28.879079 27854 leader_election.cc:288] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [CANDIDATE]: Term 1 election: Requested vote from peers 1f9f58fe8cec4e7cb448cab35a3a8a18 (127.5.118.189:36807), 69ba357d3b63493eb956d5d63e0ee60e (127.5.118.188:33941)
I20260620 14:55:28.879379 28004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18e3d719e40b4503815fddd412fac961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "69ba357d3b63493eb956d5d63e0ee60e"
I20260620 14:55:28.879479 28004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:28.880436 28004 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18e3d719e40b4503815fddd412fac961 in term 1.
W20260620 14:55:28.881315 28066 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:55:28.881434 28066 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:55:28.880196 27812 master_runner.cc:186] Error getting master registration for 127.5.118.189:36807: OK, instance_id { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" instance_seqno: 1781967328665121 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:55:28.883716 27829 leader_election.cc:302] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18e3d719e40b4503815fddd412fac961, 69ba357d3b63493eb956d5d63e0ee60e; no voters:
I20260620 14:55:28.884001 28038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:28.884212 28038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [term 1 LEADER]: Becoming Leader. State: Replica: 18e3d719e40b4503815fddd412fac961, State: Running, Role: LEADER
I20260620 14:55:28.884318 28038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } }
I20260620 14:55:28.884761 28038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18e3d719e40b4503815fddd412fac961. Latest consensus state: current_term: 1 leader_uuid: "18e3d719e40b4503815fddd412fac961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } } }
I20260620 14:55:28.884848 28038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:28.886274 27812 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:55:28.888878 28073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:28.889169 28073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:28.886612 27962 master_runner.cc:186] Error getting master registration for 127.5.118.189:36807: OK, instance_id { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" instance_seqno: 1781967328665121 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:55:28.889250 27962 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:55:28.891994 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.881429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.5.118.188:33941)
0620 14:55:28.881466 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.881468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.881710 (+ 242us) server_negotiation.cc:407] Connection header received
0620 14:55:28.881823 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.881824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.881836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.881899 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.883127 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.883850 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.885696 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.886032 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.886074 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.888530 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.888543 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.888543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.888550 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.890509 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.891017 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.891018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.891019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.891081 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.891321 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.891322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.891323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.891484 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.891595 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.891781 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.891876 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:55:28.894189 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.881239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.5.118.189:36807)
0620 14:55:28.881296 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.881298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.881705 (+ 407us) server_negotiation.cc:407] Connection header received
0620 14:55:28.881803 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.881804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.881817 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.881852 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.882712 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.883434 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.884446 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.885509 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.885567 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.890283 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.890285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.890285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.890294 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.892348 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.892985 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.892987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.892987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.893049 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.893407 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.893408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.893408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.893562 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.893633 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.894000 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.894088 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 14:55:28.894690 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.883482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.5.118.190:45519)
0620 14:55:28.883525 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.883527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.883535 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:55:28.883554 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.883555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.883563 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.883616 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.887369 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.888267 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.889331 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.889644 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.889697 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.891694 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.891696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.891696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.891704 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.893410 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.893794 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.893795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.893796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.893853 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.894034 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.894035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.894035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.894150 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.894246 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.894514 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.894594 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:55:28.901454 28004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [term 1 FOLLOWER]: Refusing update from remote peer 18e3d719e40b4503815fddd412fac961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:55:28.901892 28038 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:28.903612 28051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18e3d719e40b4503815fddd412fac961. Latest consensus state: current_term: 1 leader_uuid: "18e3d719e40b4503815fddd412fac961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } } }
I20260620 14:55:28.903716 28051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:28.904304 27955 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } attempt: 3
Jun 20 14:55:28 dist-test-slave-2m5f krb5kdc[27799](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967328, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
I20260620 14:55:28.913264 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.904566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:33941 (local address 127.0.0.1:55748)
0620 14:55:28.904616 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:28.904628 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:28.904665 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:28.904786 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:28.904787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:28.904971 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:28.905037 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.905039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.905999 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.906001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.906850 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:28.906852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.907382 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.907383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:28.907442 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.907967 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:28.907973 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:28.909638 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:28.911893 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.911894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.911895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.912135 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.912406 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:28.912406 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:28.912407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:28.912488 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:28.912892 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:28.912893 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:28.912943 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 14:55:28.913012 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:55:28.913139 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 14:55:28.913466 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:28.904609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.5.118.188:33941)
0620 14:55:28.904666 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.904668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.904677 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:55:28.904713 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.904714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.904723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.904781 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.905106 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.905894 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.906963 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:28.907294 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:28.907366 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:28.909770 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:28.909772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:28.909772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:28.909781 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:28.911803 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.912276 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.912277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.912278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.912346 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:28.912565 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:28.912566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:28.912566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:28.912795 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:28.912863 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:28.913103 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 14:55:28.913314 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 14:55:28.915572 27955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18: Bootstrap starting.
I20260620 14:55:28.916381 27955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:28.916703 27955 log.cc:824] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:28.921499 27955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18: No bootstrap required, opened a new log
I20260620 14:55:28.921437 28038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18e3d719e40b4503815fddd412fac961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } } }
I20260620 14:55:28.921551 28038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:28.921633 28074 mvcc.cc:205] Tried to move back new op lower bound from 7298938179177684992 to 7298938179110739968. Current Snapshot: MvccSnapshot[applied={T|T < 7298938179177684992}]
I20260620 14:55:28.922098 28051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18e3d719e40b4503815fddd412fac961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } } }
I20260620 14:55:28.922199 28051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:28.922907 28073 catalog_manager.cc:1347] Generated new cluster ID: f51a0f39dab34433b84cdc6c9b21dd3b
I20260620 14:55:28.922963 28073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:28.923622 27955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } }
I20260620 14:55:28.923750 27955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9f58fe8cec4e7cb448cab35a3a8a18, State: Initialized, Role: FOLLOWER
I20260620 14:55:28.923827 27955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } }
I20260620 14:55:28.924126 27955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:28.924425 27929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18e3d719e40b4503815fddd412fac961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18"
I20260620 14:55:28.924525 27929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:28.925395 27929 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18e3d719e40b4503815fddd412fac961 in term 1.
I20260620 14:55:28.924116 28080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } } }
I20260620 14:55:28.928817 28080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:28.925995 27928 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [term 1 FOLLOWER]: Refusing update from remote peer 18e3d719e40b4503815fddd412fac961: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:55:28.929339 28071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:28.932124 28080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18e3d719e40b4503815fddd412fac961. Latest consensus state: current_term: 1 leader_uuid: "18e3d719e40b4503815fddd412fac961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } } }
I20260620 14:55:28.932235 28080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:28.932266 28080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18e3d719e40b4503815fddd412fac961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } } }
I20260620 14:55:28.932296 28080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:55:28.926324 28091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:55:28.932542 28091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:55:28.935679 28093 mvcc.cc:205] Tried to move back new op lower bound from 7298938179177684992 to 7298938179110739968. Current Snapshot: MvccSnapshot[applied={T|T < 7298938179177684992}]
I20260620 14:55:28.936554 28073 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:28.937216 28073 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:28.939651 28038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18e3d719e40b4503815fddd412fac961" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3d719e40b4503815fddd412fac961" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 45519 } } peers { permanent_uuid: "1f9f58fe8cec4e7cb448cab35a3a8a18" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 36807 } } peers { permanent_uuid: "69ba357d3b63493eb956d5d63e0ee60e" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 33941 } } }
I20260620 14:55:28.939819 28038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:28.948259 28073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18e3d719e40b4503815fddd412fac961: Generated new TSK 0
I20260620 14:55:28.948518 28073 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:55:28.973630 27884 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:28.971797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51978 (local address 127.5.118.190:45519)
0620 14:55:28.971844 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.971846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.972865 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 14:55:28.972978 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.972980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.972991 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.973025 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.973549 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51978: BlockingRecv error: recv got EOF from 127.0.0.1:51978 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 14:55:28.974427 27959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:28.972254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56904 (local address 127.5.118.189:36807)
0620 14:55:28.972295 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.972297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.973882 (+ 1585us) server_negotiation.cc:407] Connection header received
0620 14:55:28.973980 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.973981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.973992 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.974023 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.974360 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56904: BlockingRecv error: recv got EOF from 127.0.0.1:56904 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 14:55:28.975221 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:28.972567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.5.118.188:33941)
0620 14:55:28.972621 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.972623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.974690 (+ 2067us) server_negotiation.cc:407] Connection header received
0620 14:55:28.974790 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.974791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.974802 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.974833 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.975156 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55764: BlockingRecv error: recv got EOF from 127.0.0.1:55764 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 14:55:28.978099 27959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:28.977161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.5.118.189:36807)
0620 14:55:28.977211 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.977214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.977487 (+ 273us) server_negotiation.cc:407] Connection header received
0620 14:55:28.977596 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.977597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.977634 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.977665 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.978031 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56920: BlockingRecv error: recv got EOF from 127.0.0.1:56920 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:55:28.981299 27959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:28.980525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.5.118.189:36807)
0620 14:55:28.980560 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.980562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.980801 (+ 239us) server_negotiation.cc:407] Connection header received
0620 14:55:28.980893 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.980894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.980904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.980931 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.981252 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56926: BlockingRecv error: recv got EOF from 127.0.0.1:56926 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 14:55:28.985334 27959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:28.984546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.5.118.189:36807)
0620 14:55:28.984582 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.984584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.984821 (+ 237us) server_negotiation.cc:407] Connection header received
0620 14:55:28.984913 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.984914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.984924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.984956 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.985282 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56934: BlockingRecv error: recv got EOF from 127.0.0.1:56934 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:55:28.991465 27959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:28.990626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56950 (local address 127.5.118.189:36807)
0620 14:55:28.990666 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:28.990668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:28.990918 (+ 250us) server_negotiation.cc:407] Connection header received
0620 14:55:28.991018 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:28.991019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:28.991029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:28.991059 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:28.991411 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56950: BlockingRecv error: recv got EOF from 127.0.0.1:56950 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:55:29.002009 27959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:29.001000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.5.118.189:36807)
0620 14:55:29.001046 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:29.001048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:29.001350 (+ 302us) server_negotiation.cc:407] Connection header received
0620 14:55:29.001463 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:29.001464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:29.001477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:29.001508 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:29.001946 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56960: BlockingRecv error: recv got EOF from 127.0.0.1:56960 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:55:29.020593 27959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:29.019530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.5.118.189:36807)
0620 14:55:29.019577 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:29.019580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:29.019877 (+ 297us) server_negotiation.cc:407] Connection header received
0620 14:55:29.020007 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:29.020008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:29.020021 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:29.020052 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:29.020525 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56976: BlockingRecv error: recv got EOF from 127.0.0.1:56976 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:55:29.055115 27959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:29.054149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.5.118.189:36807)
0620 14:55:29.054194 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:29.054196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:29.054491 (+ 295us) server_negotiation.cc:407] Connection header received
0620 14:55:29.054604 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:29.054605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:29.054617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:29.054649 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:29.055051 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56992: BlockingRecv error: recv got EOF from 127.0.0.1:56992 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 14:55:29.121783 27959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:29.120880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.5.118.189:36807)
0620 14:55:29.120949 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:29.120951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:29.121094 (+ 143us) server_negotiation.cc:407] Connection header received
0620 14:55:29.121271 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:29.121272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:29.121283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:29.121360 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:29.121716 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57002: BlockingRecv error: recv got EOF from 127.0.0.1:57002 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 14:55:29.252084 27959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:29.251273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.5.118.189:36807)
0620 14:55:29.251373 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:29.251376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:29.251492 (+ 116us) server_negotiation.cc:407] Connection header received
0620 14:55:29.251581 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:29.251582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:29.251594 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:29.251674 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:29.252027 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57006: BlockingRecv error: recv got EOF from 127.0.0.1:57006 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 14:55:29.510433 27959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:29.509598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.5.118.189:36807)
0620 14:55:29.509683 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:29.509685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:29.509773 (+ 88us) server_negotiation.cc:407] Connection header received
0620 14:55:29.509802 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:29.509803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:29.509814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:29.509860 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:29.510358 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42864: BlockingRecv error: recv got EOF from 127.0.0.1:42864 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:55:29.881953 28066 catalog_manager.cc:1259] Loaded cluster ID: f51a0f39dab34433b84cdc6c9b21dd3b
I20260620 14:55:29.882032 28066 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e: loading cluster ID for follower catalog manager: success
I20260620 14:55:29.882967 28066 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e: acquiring CA information for follower catalog manager: success
I20260620 14:55:29.883332 28066 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69ba357d3b63493eb956d5d63e0ee60e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:55:29.933091 28091 catalog_manager.cc:1259] Loaded cluster ID: f51a0f39dab34433b84cdc6c9b21dd3b
I20260620 14:55:29.933210 28091 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18: loading cluster ID for follower catalog manager: success
I20260620 14:55:29.934109 28091 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18: acquiring CA information for follower catalog manager: success
I20260620 14:55:29.934476 28091 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f9f58fe8cec4e7cb448cab35a3a8a18: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:55:30.027068 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:30.023656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.5.118.189:36807)
0620 14:55:30.023789 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:30.023791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:30.023918 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:55:30.023956 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:30.023957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:30.024010 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:30.024105 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:30.024572 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:30.025212 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:30.026147 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:30.026437 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:30.026509 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:30.026700 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:30.026821 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:30.026892 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:30.026968 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 14:55:30.026988 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260620 14:55:30.032258 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:30.029483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.5.118.188:33941)
0620 14:55:30.029611 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:30.029614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:30.029623 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:55:30.029757 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:30.029758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:30.029803 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:30.029905 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:30.030042 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:30.030578 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:30.031366 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:30.031594 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:30.031685 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:30.031859 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:30.031977 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:30.032073 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:30.032113 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 14:55:30.032135 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260620 14:55:30.037176 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:30.034239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.5.118.189:36807)
0620 14:55:30.034359 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:30.034361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:30.034385 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:55:30.034482 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:30.034483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:30.034496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:30.034560 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:30.034739 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:30.035519 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:30.036442 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:30.036662 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:30.036753 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:30.036911 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:30.036945 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:30.036992 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:30.037050 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 14:55:30.037073 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 14:55:30.037616 28175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:30.034179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.5.118.190:45519)
0620 14:55:30.034292 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:30.034294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:30.034447 (+ 153us) server_negotiation.cc:407] Connection header received
0620 14:55:30.034494 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:30.034495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:30.034579 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:30.034660 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:30.034825 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:30.035430 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:30.036496 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:30.036833 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:30.036890 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:30.037123 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:30.037260 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:30.037327 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:30.037493 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 14:55:30.037521 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
I20260620 14:55:30.037685 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:30.034592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51744 (local address 127.5.118.188:33941)
0620 14:55:30.034630 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:30.034632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:30.034752 (+ 120us) server_negotiation.cc:407] Connection header received
0620 14:55:30.034860 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:30.034861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:30.034873 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:30.034924 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:30.035081 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:30.035767 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:30.036780 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:30.037100 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:30.037145 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:30.037350 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:30.037385 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:30.037414 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:30.037552 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:55:30.037574 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:55:30.039909 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 27812
I20260620 14:55:30.045475 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 27887
I20260620 14:55:30.050480 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 27962
2026-06-20T14:55:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2895 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:55:30 dist-test-slave-2m5f krb5kdc[28180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:55:30 dist-test-slave-2m5f krb5kdc[28180](info): set up 2 sockets
Jun 20 14:55:30 dist-test-slave-2m5f krb5kdc[28180](info): commencing operation
krb5kdc: starting...
W20260620 14:55:31.116653 5594 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/test-user.keytab.
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:55:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:55:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.118.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
Entry for principal kudu/127.5.118.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab.
I20260620 14:55:31.244771 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:37957
--webserver_interface=127.5.118.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:37957,127.5.118.189:43167,127.5.118.188:44495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:55:31.328364 28193 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:55:31.328585 28193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:31.328603 28193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:31.328625 28193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:31.329747 28193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:31.329784 28193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:31.329798 28193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:31.329811 28193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:31.329823 28193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:31.331233 28193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:37957,127.5.118.189:43167,127.5.118.188:44495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.190.keytab
--principal=kudu/127.5.118.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.190:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.118.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:31.331439 28193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:31.331619 28193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:31.336678 28198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:31.336678 28199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:31.336678 28201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:31.336758 28193 server_base.cc:1034] running on GCE node
I20260620 14:55:31.337196 28193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:31.337553 28193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:31.338743 28193 hybrid_clock.cc:648] HybridClock initialized: now 1781967331338735 us; error 54 us; skew 500 ppm
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:31.341740 28193 init.cc:377] Logged in from keytab as kudu/127.5.118.190@KRBTEST.COM (short username kudu)
I20260620 14:55:31.342245 28193 webserver.cc:460] Webserver started at http://127.5.118.190:41067/ using document root <none> and password file <none>
I20260620 14:55:31.342531 28193 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:31.342618 28193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:31.342774 28193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:31.343873 28193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/data/instance:
uuid: "838c98fa1463461199e4e98d82c671c3"
format_stamp: "Formatted at 2026-06-20 14:55:31 on dist-test-slave-2m5f"
server_key: "adf9225ad04fc72a7c3af5b4b4674cb0"
server_key_iv: "10ecf8e0835efac0c77878098f37c8a5"
server_key_version: "encryptionkey@0"
I20260620 14:55:31.344321 28193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/wal/instance:
uuid: "838c98fa1463461199e4e98d82c671c3"
format_stamp: "Formatted at 2026-06-20 14:55:31 on dist-test-slave-2m5f"
server_key: "adf9225ad04fc72a7c3af5b4b4674cb0"
server_key_iv: "10ecf8e0835efac0c77878098f37c8a5"
server_key_version: "encryptionkey@0"
I20260620 14:55:31.345717 28193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:31.346522 28208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:31.346729 28193 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:31.346837 28193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/wal
uuid: "838c98fa1463461199e4e98d82c671c3"
format_stamp: "Formatted at 2026-06-20 14:55:31 on dist-test-slave-2m5f"
server_key: "adf9225ad04fc72a7c3af5b4b4674cb0"
server_key_iv: "10ecf8e0835efac0c77878098f37c8a5"
server_key_version: "encryptionkey@0"
I20260620 14:55:31.346925 28193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:31.372340 28193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:31.374545 28193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:31.374743 28193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:31.378983 28193 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.190:37957
I20260620 14:55:31.379024 28260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.190:37957 every 8 connection(s)
I20260620 14:55:31.379422 28193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/data/info.pb
I20260620 14:55:31.380169 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 28193
I20260620 14:55:31.380335 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-0/wal/instance
I20260620 14:55:31.380612 5594 external_mini_cluster.cc:1428] Setting key 87d30870fa65ed005610df9e9e4d669a
I20260620 14:55:31.380616 28261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:31.382236 28261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.118.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.189@KRBTEST.COM" created.
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:31.395634 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.383026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.5.118.190:37957)
0620 14:55:31.383188 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.383199 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.383324 (+ 125us) server_negotiation.cc:407] Connection header received
0620 14:55:31.383886 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.383889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.383910 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.383997 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.384641 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.385459 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.386382 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.386715 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.386851 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.391981 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.391992 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.391992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.392001 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.393832 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.394325 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.394329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.394329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.394418 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.394661 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.394662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.394662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.394901 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.395011 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.395206 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.395371 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260620 14:55:31.395637 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.382984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:37957 (local address 127.0.0.1:44456)
0620 14:55:31.383171 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.383213 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:31.383714 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:31.384018 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:31.384021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:31.384375 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:31.384534 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.384539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.385580 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.385581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.386280 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.386283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.386839 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.386841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.386910 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.387535 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:31.387543 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:31.391859 (+ 4316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:31.393992 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.394000 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.394002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.394212 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.394497 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.394498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.394499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.394587 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.394988 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:31.394993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:31.395060 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 14:55:31.395171 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:55:31.395443 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":110,"thread_start_us":78,"threads_started":1}
I20260620 14:55:31.396742 28261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:55:31.397181 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.397017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:43167 (local address 127.0.0.1:49330)
0620 14:55:31.397085 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.397146 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:55:31.397351 28209 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.118.189:43167: connect: Connection refused (error 111)
I20260620 14:55:31.397827 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.397692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:43167 (local address 127.0.0.1:49346)
0620 14:55:31.397764 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.397795 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:55:31.398017 28261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.189:43167: Network error: Client connection negotiation failed: client connection to 127.5.118.189:43167: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab.
Entry for principal kudu/127.5.118.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab.
I20260620 14:55:31.407680 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.189:43167
--webserver_interface=127.5.118.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab
--principal=kudu/127.5.118.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:37957,127.5.118.189:43167,127.5.118.188:44495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:55:31.447274 28261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } attempt: 1
I20260620 14:55:31.447819 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.447681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:43167 (local address 127.0.0.1:49352)
0620 14:55:31.447748 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.447789 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:55:31.448225 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.448090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:43167 (local address 127.0.0.1:49368)
0620 14:55:31.448142 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.448167 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:55:31.448330 28261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.189:43167: Network error: Client connection negotiation failed: client connection to 127.5.118.189:43167: connect: Connection refused (error 111)
W20260620 14:55:31.490713 28268 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:55:31.491016 28268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:31.491057 28268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:31.491088 28268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:31.492303 28268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:31.492393 28268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:31.492416 28268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:31.492436 28268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:31.492462 28268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:31.494109 28268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:37957,127.5.118.189:43167,127.5.118.188:44495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.189.keytab
--principal=kudu/127.5.118.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.189:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.118.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:31.494428 28268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:31.494683 28268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:31.499856 28273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:31.499856 28276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:31.499903 28268 server_base.cc:1034] running on GCE node
W20260620 14:55:31.499998 28274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:31.500375 28268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:31.500741 28268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:31.501945 28268 hybrid_clock.cc:648] HybridClock initialized: now 1781967331501931 us; error 53 us; skew 500 ppm
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:31.504753 28268 init.cc:377] Logged in from keytab as kudu/127.5.118.189@KRBTEST.COM (short username kudu)
I20260620 14:55:31.505254 28268 webserver.cc:460] Webserver started at http://127.5.118.189:36149/ using document root <none> and password file <none>
I20260620 14:55:31.505465 28268 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:31.505509 28268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:31.505602 28268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:31.506671 28268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/data/instance:
uuid: "2df243838ef14e098fd3251069ccf4d1"
format_stamp: "Formatted at 2026-06-20 14:55:31 on dist-test-slave-2m5f"
server_key: "870fa6aff9806c988be74e6d5b74a719"
server_key_iv: "1a9623b882c39e9ea5ce3bb082e08a1e"
server_key_version: "encryptionkey@0"
I20260620 14:55:31.507000 28268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/wal/instance:
uuid: "2df243838ef14e098fd3251069ccf4d1"
format_stamp: "Formatted at 2026-06-20 14:55:31 on dist-test-slave-2m5f"
server_key: "870fa6aff9806c988be74e6d5b74a719"
server_key_iv: "1a9623b882c39e9ea5ce3bb082e08a1e"
server_key_version: "encryptionkey@0"
I20260620 14:55:31.508431 28268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 14:55:31.509310 28283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:31.509524 28268 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:31.509611 28268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/wal
uuid: "2df243838ef14e098fd3251069ccf4d1"
format_stamp: "Formatted at 2026-06-20 14:55:31 on dist-test-slave-2m5f"
server_key: "870fa6aff9806c988be74e6d5b74a719"
server_key_iv: "1a9623b882c39e9ea5ce3bb082e08a1e"
server_key_version: "encryptionkey@0"
I20260620 14:55:31.509682 28268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:31.516498 28261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } attempt: 2
I20260620 14:55:31.516976 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.516808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:43167 (local address 127.0.0.1:49370)
0620 14:55:31.516878 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.516936 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 14:55:31.517478 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.517383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:43167 (local address 127.0.0.1:49384)
0620 14:55:31.517432 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.517460 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.189:43167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 14:55:31.517609 28261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.189:43167: Network error: Client connection negotiation failed: client connection to 127.5.118.189:43167: connect: Connection refused (error 111)
I20260620 14:55:31.524413 28268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:31.526569 28268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:31.526787 28268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:31.530995 28268 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.189:43167
I20260620 14:55:31.531090 28335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.189:43167 every 8 connection(s)
I20260620 14:55:31.531471 28268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/data/info.pb
I20260620 14:55:31.532618 28336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:31.532894 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 28268
I20260620 14:55:31.533006 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-1/wal/instance
I20260620 14:55:31.533231 5594 external_mini_cluster.cc:1428] Setting key ad258c85d3aa46b2a1cd6447715e8d33
I20260620 14:55:31.534523 28336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.118.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.118.188@KRBTEST.COM" created.
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:31.548722 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.535492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.5.118.190:37957)
0620 14:55:31.535591 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.535593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.535815 (+ 222us) server_negotiation.cc:407] Connection header received
0620 14:55:31.536440 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.536442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.536459 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.536531 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.537201 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.538050 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.539126 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.539483 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.539569 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.545120 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.545122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.545123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.545132 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.547085 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.547613 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.547615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.547615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.547680 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.547970 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.547971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.547971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.548105 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.548211 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.548530 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.548622 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:55:31.548941 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.535412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:37957 (local address 127.0.0.1:44466)
0620 14:55:31.535675 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.535712 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:31.536330 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:31.536612 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:31.536616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:31.536882 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:31.537093 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.537098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.538173 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.538173 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.539019 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.539022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.539597 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.539598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.539714 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.540324 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:31.540332 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:31.544949 (+ 4617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:31.547188 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.547202 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.547205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.547507 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.547754 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.547755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.547756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.547886 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.548212 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:31.548221 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:31.548375 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 14:55:31.548510 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 14:55:31.548691 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":132,"threads_started":1}
I20260620 14:55:31.549537 28336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
I20260620 14:55:31.558974 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.549862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:43167 (local address 127.0.0.1:49394)
0620 14:55:31.549929 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.549943 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:31.549998 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:31.550235 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:31.550236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:31.550417 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:31.550489 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.550490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.551653 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.551655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.552564 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.552567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.553046 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.553047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.553106 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.553673 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:31.553678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:31.556007 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:31.557864 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.557865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.557865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.558056 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.558290 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.558292 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.558292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.558368 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.558686 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:31.558686 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:31.558725 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:55:31.558778 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 14:55:31.558894 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:55:31.559162 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.549899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.5.118.189:43167)
0620 14:55:31.550068 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.550073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.550093 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:55:31.550116 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.550119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.550136 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.550227 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.550567 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.551550 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.552654 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.552966 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.553022 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.556096 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.556103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.556104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.556110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.557724 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.558140 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.558142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.558143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.558197 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.558466 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.558467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.558467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.558619 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.558688 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.558856 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.559023 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":139,"thread_start_us":113,"threads_started":1}
I20260620 14:55:31.559952 28336 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:55:31.560434 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.560282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58738)
0620 14:55:31.560337 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.560413 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 14:55:31.560621 28286 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
I20260620 14:55:31.561127 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.561033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58742)
0620 14:55:31.561083 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.561110 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:55:31.561266 28336 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:44495: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.118.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab.
Entry for principal kudu/127.5.118.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab.
I20260620 14:55:31.564308 5594 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjla6OK/build/release/bin/kudu
/tmp/dist-test-taskjla6OK/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.188:44495
--webserver_interface=127.5.118.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab
--principal=kudu/127.5.118.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.118.148:43459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.118.190:37957,127.5.118.189:43167,127.5.118.188:44495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:55:31.608798 28261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } attempt: 3
I20260620 14:55:31.610548 28336 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } attempt: 1
I20260620 14:55:31.610986 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.610861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58758)
0620 14:55:31.610920 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.610958 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 14:55:31.611346 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.611249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58764)
0620 14:55:31.611300 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.611325 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:55:31.611466 28336 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:44495: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
I20260620 14:55:31.616815 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.609315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.5.118.189:43167)
0620 14:55:31.609388 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.609391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.609403 (+ 12us) server_negotiation.cc:407] Connection header received
0620 14:55:31.609505 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.609506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.609516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.609598 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.609942 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.610454 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.611558 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.611793 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.611857 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.614088 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.614090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.614091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.614099 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.615557 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.615954 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.615955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.615956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.616013 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.616272 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.616273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.616273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.616399 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.616485 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.616643 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.616721 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:55:31.616818 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.609237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:43167 (local address 127.0.0.1:49410)
0620 14:55:31.609301 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.609311 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:31.609415 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:31.609604 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:31.609605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:31.609783 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:31.609849 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.609851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.610604 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.610605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.611448 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.611451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.611901 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.611902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.611987 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.612544 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:31.612550 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:31.613989 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:31.615669 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.615675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.615675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.615853 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.616085 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.616085 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.616086 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.616147 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.616476 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:31.616477 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:31.616525 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:55:31.616619 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 14:55:31.616721 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:55:31.617314 28261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:55:31.617714 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.617587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58776)
0620 14:55:31.617641 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.617681 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:55:31.618121 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.618008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58790)
0620 14:55:31.618060 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.618094 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 14:55:31.618219 28261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:44495: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
I20260620 14:55:31.649394 28261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } attempt: 1
I20260620 14:55:31.649919 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.649777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58792)
0620 14:55:31.649844 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.649891 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:55:31.650390 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.650258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58798)
0620 14:55:31.650310 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.650335 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:55:31.650489 28261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:44495: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
W20260620 14:55:31.653535 28343 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:55:31.653738 28343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:55:31.653756 28343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:55:31.653815 28343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:55:31.654971 28343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:55:31.655017 28343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:55:31.655032 28343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:55:31.655124 28343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:55:31.655148 28343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:55:31.656795 28343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.118.148:43459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.118.190:37957,127.5.118.189:43167,127.5.118.188:44495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/krb5kdc/kudu_127.5.118.188.keytab
--principal=kudu/127.5.118.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.118.188:44495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.118.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 13:54:59 UTC on ccycloud.quasar-mqrrzv.root.comops.site
build id 4457505
I20260620 14:55:31.657089 28343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:55:31.657342 28343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:55:31.662568 28351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:31.662618 28348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:55:31.662578 28349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:55:31.662637 28343 server_base.cc:1034] running on GCE node
I20260620 14:55:31.663151 28343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:55:31.663498 28343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:55:31.664677 28343 hybrid_clock.cc:648] HybridClock initialized: now 1781967331664650 us; error 39 us; skew 500 ppm
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:55:31.667387 28343 init.cc:377] Logged in from keytab as kudu/127.5.118.188@KRBTEST.COM (short username kudu)
I20260620 14:55:31.668002 28343 webserver.cc:460] Webserver started at http://127.5.118.188:38157/ using document root <none> and password file <none>
I20260620 14:55:31.668321 28343 fs_manager.cc:359] Metadata directory not provided
I20260620 14:55:31.668403 28343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:55:31.668565 28343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:55:31.669656 28343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/data/instance:
uuid: "e219a0f031484f41933f1881409cdec7"
format_stamp: "Formatted at 2026-06-20 14:55:31 on dist-test-slave-2m5f"
server_key: "fab11c11e1c543d7e98a76ff8e591703"
server_key_iv: "f8455201ca0a898ea243e89f9d624fca"
server_key_version: "encryptionkey@0"
I20260620 14:55:31.670053 28343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/wal/instance:
uuid: "e219a0f031484f41933f1881409cdec7"
format_stamp: "Formatted at 2026-06-20 14:55:31 on dist-test-slave-2m5f"
server_key: "fab11c11e1c543d7e98a76ff8e591703"
server_key_iv: "f8455201ca0a898ea243e89f9d624fca"
server_key_version: "encryptionkey@0"
I20260620 14:55:31.671428 28343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:55:31.672312 28358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:55:31.672616 28343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 14:55:31.672768 28343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/data,/tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/wal
uuid: "e219a0f031484f41933f1881409cdec7"
format_stamp: "Formatted at 2026-06-20 14:55:31 on dist-test-slave-2m5f"
server_key: "fab11c11e1c543d7e98a76ff8e591703"
server_key_iv: "f8455201ca0a898ea243e89f9d624fca"
server_key_version: "encryptionkey@0"
I20260620 14:55:31.672883 28343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:55:31.679611 28336 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } attempt: 2
I20260620 14:55:31.680094 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.679940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58804)
0620 14:55:31.680015 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.680065 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:55:31.680490 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.680383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58810)
0620 14:55:31.680435 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.680460 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 14:55:31.680581 28336 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.118.188:44495: Network error: Client connection negotiation failed: client connection to 127.5.118.188:44495: connect: Connection refused (error 111)
I20260620 14:55:31.691494 28343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:55:31.693696 28343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:55:31.693897 28343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:55:31.698505 28343 rpc_server.cc:307] RPC server started. Bound to: 127.5.118.188:44495
I20260620 14:55:31.698563 28410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.118.188:44495 every 8 connection(s)
I20260620 14:55:31.698975 28343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/data/info.pb
I20260620 14:55:31.699450 5594 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjla6OK/build/release/bin/kudu as pid 28343
I20260620 14:55:31.699604 5594 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjla6OK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967053866185-5594-0/minicluster-data/master-2/wal/instance
I20260620 14:55:31.699920 5594 external_mini_cluster.cc:1428] Setting key d09b363bcbef69fdc3a05cd5a4733d29
I20260620 14:55:31.700373 28411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:55:31.702595 28411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:31.711549 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.703333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.5.118.190:37957)
0620 14:55:31.703390 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.703393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.703543 (+ 150us) server_negotiation.cc:407] Connection header received
0620 14:55:31.703637 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.703638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.703650 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.703707 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.704220 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.704749 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.705718 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.706057 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.706142 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.708519 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.708520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.708520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.708528 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.710032 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.710515 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.710516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.710516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.710569 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.710847 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.710847 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.710848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.710982 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.711077 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.711301 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.711379 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 14:55:31.711687 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.703247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.190:37957 (local address 127.0.0.1:44470)
0620 14:55:31.703399 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.703449 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:31.703564 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:31.703724 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:31.703727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:31.704017 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:31.704091 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.704093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.704861 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.704862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.705620 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.705622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.706194 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.706195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.706255 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.706813 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:31.706821 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:31.708407 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:31.710125 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.710131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.710133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.710437 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.710652 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.710653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.710653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.710775 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.711089 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:31.711094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:31.711160 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 14:55:31.711250 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:55:31.711481 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":115,"thread_start_us":83,"threads_started":1}
I20260620 14:55:31.711697 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.700691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.5.118.188:44495)
0620 14:55:31.700892 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.700900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.700926 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:55:31.701590 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.701597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.701650 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.701818 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.702485 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.703720 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.704593 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.705058 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.705186 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.707847 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.707854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.707854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.707882 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.709844 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.710258 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.710261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.710261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.710346 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.710652 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.710653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.710653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.710969 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.711105 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.711219 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.711394 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260620 14:55:31.712846 28411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.190@KRBTEST.COM
I20260620 14:55:31.720956 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.713127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.189:43167 (local address 127.0.0.1:49412)
0620 14:55:31.713224 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.713235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:31.713310 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:31.713462 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:31.713463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:31.713614 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:31.713671 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.713673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.714589 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.714590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.715395 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.715397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.715906 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.715908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.715983 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.716522 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:31.716527 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:31.717920 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:31.719786 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.719787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.719788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.719967 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.720228 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.720229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.720230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.720291 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.720607 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:31.720608 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:31.720647 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:55:31.720703 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 14:55:31.720824 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 14:55:31.720959 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.713217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.5.118.189:43167)
0620 14:55:31.713294 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.713296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.713306 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:55:31.713392 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.713393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.713405 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.713458 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.713750 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.714496 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.715500 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.715822 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.715892 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.718015 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.718016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.718016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.718024 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.719675 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.720062 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.720064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.720064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.720127 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.720363 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.720365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.720365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.720525 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.720601 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.720756 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.720839 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:55:31.721661 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.712750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.5.118.190:37957)
0620 14:55:31.712832 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.712834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.712843 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:55:31.712863 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.712863 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.712873 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.712935 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.713643 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.714489 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.715499 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.715796 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.715874 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.718712 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.718714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.718714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.718721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.720372 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.720777 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.720778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.720779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.720837 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.721070 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.721071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.721071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.721213 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.721373 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.721488 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.721562 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:55:31.721763 28411 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.188@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
I20260620 14:55:31.725649 28261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } attempt: 2
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.118.189@KRBTEST.COM
I20260620 14:55:31.729756 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.722048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58816)
0620 14:55:31.722110 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.722121 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:31.722165 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:31.722419 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:31.722420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:31.722557 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:31.722584 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.722585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.723248 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.723249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.723724 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.723725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.724115 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.724115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.724163 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.724631 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:31.724636 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:31.725861 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:31.727704 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.727705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.727706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.727931 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.728838 (+ 907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.728839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.728840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.728909 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.729150 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:31.729151 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:31.729189 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:55:31.729258 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 14:55:31.729647 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 14:55:31.729990 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.722128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.5.118.188:44495)
0620 14:55:31.722164 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.722166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.722173 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:55:31.722214 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.722214 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.722222 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.722418 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.722663 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.723136 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.723811 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.724024 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.724115 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.725974 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.725975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.725976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.725982 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.727631 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.728690 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.728691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.728691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.728752 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.728970 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.728970 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.728971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.729077 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.729160 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.729313 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.729922 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":522,"server-negotiator.queue_time_us":19}
I20260620 14:55:31.730450 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.722306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49418 (local address 127.5.118.189:43167)
0620 14:55:31.722343 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.722345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.722353 (+ 8us) server_negotiation.cc:407] Connection header received
0620 14:55:31.722401 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.722402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.722412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.722467 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.722936 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.723418 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.724111 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.724462 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.724529 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.727158 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.727159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.727159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.727166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.728758 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.729181 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.729182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.729182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.729243 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.729498 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.729499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.729500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.729648 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.730205 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.730253 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.730314 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.190@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
I20260620 14:55:31.732095 28411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7: Bootstrap starting.
I20260620 14:55:31.732550 5594 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:55:31.732911 28411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:31.733265 28411 log.cc:824] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:31.734057 28411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7: No bootstrap required, opened a new log
I20260620 14:55:31.733851 28235 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "838c98fa1463461199e4e98d82c671c3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44480
I20260620 14:55:31.735234 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.725925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58820)
0620 14:55:31.725977 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.725988 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:31.726024 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:31.726350 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:31.726351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:31.726512 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:31.726562 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.726564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.727971 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.727972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.728571 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.728573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.729094 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.729095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.729158 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.729577 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:31.729581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:31.731016 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:31.734043 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.734044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.734044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.734290 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.734514 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.734515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.734515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.734571 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.734930 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:31.734931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:31.734979 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:55:31.735031 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 14:55:31.735131 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:55:31.735592 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.725980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.5.118.188:44495)
0620 14:55:31.726230 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.726231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.726236 (+ 5us) server_negotiation.cc:407] Connection header received
0620 14:55:31.726250 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.726250 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.726258 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.726371 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.726649 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.727885 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.728672 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.728996 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.729304 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.731212 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.731214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.731214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.731221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.733939 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.734386 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.734387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.734388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.734445 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.734641 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.734641 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.734642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.734798 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.735352 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.735396 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.735476 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":93,"threads_started":1}
I20260620 14:55:31.735873 28411 raft_consensus.cc:348] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } }
I20260620 14:55:31.736048 28411 raft_consensus.cc:729] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e219a0f031484f41933f1881409cdec7, State: Initialized, Role: FOLLOWER
I20260620 14:55:31.736140 28411 consensus_queue.cc:260] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } }
I20260620 14:55:31.736460 28418 sys_catalog.cc:455] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } } }
I20260620 14:55:31.736524 28411 sys_catalog.cc:564] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:31.736554 28418 sys_catalog.cc:458] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:31.737112 28261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3: Bootstrap starting.
I20260620 14:55:31.737931 28261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:31.738247 28261 log.cc:824] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3: Log is configured to *not* fsync() on all Append() calls
W20260620 14:55:31.738960 28429 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:55:31.739020 28429 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:55:31.740942 28261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3: No bootstrap required, opened a new log
I20260620 14:55:31.738806 28343 master_runner.cc:186] Error getting master registration for 127.5.118.190:37957: OK, instance_id { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" instance_seqno: 1781967331377813 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:55:31.741250 28343 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:55:31.742091 28261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } }
I20260620 14:55:31.742208 28261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838c98fa1463461199e4e98d82c671c3, State: Initialized, Role: FOLLOWER
I20260620 14:55:31.742321 28261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } }
I20260620 14:55:31.742624 28432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } } }
I20260620 14:55:31.742707 28432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:31.742718 28261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:31.742805 28235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:55:31.742862 28235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:31.743480 28235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } }
I20260620 14:55:31.743752 28235 leader_election.cc:288] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [CANDIDATE]: Term 1 election: Requested vote from peers 2df243838ef14e098fd3251069ccf4d1 (127.5.118.189:43167), e219a0f031484f41933f1881409cdec7 (127.5.118.188:44495)
I20260620 14:55:31.744052 28385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "838c98fa1463461199e4e98d82c671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e219a0f031484f41933f1881409cdec7"
I20260620 14:55:31.744246 28385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:31.745267 28385 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 838c98fa1463461199e4e98d82c671c3 in term 1.
I20260620 14:55:31.745544 28211 leader_election.cc:302] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 838c98fa1463461199e4e98d82c671c3, e219a0f031484f41933f1881409cdec7; no voters:
I20260620 14:55:31.745669 28432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:55:31.745810 28432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [term 1 LEADER]: Becoming Leader. State: Replica: 838c98fa1463461199e4e98d82c671c3, State: Running, Role: LEADER
I20260620 14:55:31.746055 28432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } }
I20260620 14:55:31.746639 28448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 838c98fa1463461199e4e98d82c671c3. Latest consensus state: current_term: 1 leader_uuid: "838c98fa1463461199e4e98d82c671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } } }
I20260620 14:55:31.746728 28448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [sys.catalog]: This master's current role is: LEADER
W20260620 14:55:31.747834 28447 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:55:31.747951 28447 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:55:31.747594 28193 master_runner.cc:186] Error getting master registration for 127.5.118.189:43167: OK, instance_id { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" instance_seqno: 1781967331529867 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:55:31.750010 28193 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:55:31.754495 28450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:55:31.757150 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.747124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.5.118.189:43167)
0620 14:55:31.747233 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.747236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.748119 (+ 883us) server_negotiation.cc:407] Connection header received
0620 14:55:31.748272 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.748273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.748285 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.748337 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.749133 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.749907 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.750946 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.751255 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.751296 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.753161 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.753163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.753164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.753172 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.755160 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.755605 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.755607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.755607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.755663 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.755893 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.755895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.755895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.756038 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.756650 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.756906 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.757025 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:55:31.758040 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.747967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.5.118.188:44495)
0620 14:55:31.748021 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.748023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.748599 (+ 576us) server_negotiation.cc:407] Connection header received
0620 14:55:31.748694 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.748695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.748708 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.748741 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.749470 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.750191 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.751120 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.751562 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.751952 (+ 390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.753871 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.753873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.753873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.753882 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.755995 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.756746 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.756747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.756748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.756812 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.757337 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.757338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.757338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.757505 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.757613 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.757838 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.757937 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:55:31.758924 28450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:55:31.759821 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.747573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.5.118.190:37957)
0620 14:55:31.750086 (+ 2513us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.750088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.750098 (+ 10us) server_negotiation.cc:407] Connection header received
0620 14:55:31.750118 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.750119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.750130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.750192 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.751118 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.751959 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.753555 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.753874 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.753916 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.756007 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.756009 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.756009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.756017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.757733 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.758780 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.758781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.758782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.758837 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.759069 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.759070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.759070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.759202 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.759270 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.759646 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.759729 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2491}
I20260620 14:55:31.761066 28385 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [term 1 FOLLOWER]: Refusing update from remote peer 838c98fa1463461199e4e98d82c671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:55:31.761404 28432 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:31.762792 28418 sys_catalog.cc:455] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 838c98fa1463461199e4e98d82c671c3. Latest consensus state: current_term: 1 leader_uuid: "838c98fa1463461199e4e98d82c671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } } }
I20260620 14:55:31.762916 28418 sys_catalog.cc:458] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:31.771721 28336 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } attempt: 3
I20260620 14:55:31.773656 28432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "838c98fa1463461199e4e98d82c671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } } }
I20260620 14:55:31.773761 28432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:31.774374 28418 sys_catalog.cc:455] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "838c98fa1463461199e4e98d82c671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } } }
I20260620 14:55:31.774531 28418 sys_catalog.cc:458] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 14:55:31 dist-test-slave-2m5f krb5kdc[28180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967331, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.118.189@KRBTEST.COM for kudu/127.5.118.188@KRBTEST.COM
I20260620 14:55:31.780246 28456 mvcc.cc:205] Tried to move back new op lower bound from 7298938190890266624 to 7298938190832480256. Current Snapshot: MvccSnapshot[applied={T|T < 7298938190890266624}]
I20260620 14:55:31.780545 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.772037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.118.188:44495 (local address 127.0.0.1:58836)
0620 14:55:31.772079 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 14:55:31.772092 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 14:55:31.772136 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:55:31.772294 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:55:31.772295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:55:31.772490 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:55:31.772545 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.772547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.773474 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.773475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.774225 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:55:31.774228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.775077 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.775078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:55:31.775139 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.775691 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:55:31.775698 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:55:31.777298 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:55:31.779337 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.779338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.779339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.779633 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.779833 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:55:31.779834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:55:31.779834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:55:31.779912 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:55:31.780204 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:55:31.780205 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:55:31.780268 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 14:55:31.780333 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 14:55:31.780450 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 14:55:31.780544 28416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:31.772101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58836 (local address 127.5.118.188:44495)
0620 14:55:31.772141 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.772144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.772162 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:55:31.772226 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.772227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.772236 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.772280 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.772623 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.773370 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.774618 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:31.774951 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:31.775017 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:31.777395 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:55:31.777397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:55:31.777397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:55:31.777405 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:55:31.779248 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.779707 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.779709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.779709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.779784 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:55:31.779980 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:55:31.779981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:55:31.779981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:55:31.780115 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:55:31.780195 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:31.780382 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:55:31.780455 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 14:55:31.780723 28450 catalog_manager.cc:1347] Generated new cluster ID: 9a0a07e9ba794afbbb25e73cc1c53a3e
I20260620 14:55:31.780769 28450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:55:31.781996 28336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1: Bootstrap starting.
I20260620 14:55:31.782691 28336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1: Neither blocks nor log segments found. Creating new log.
I20260620 14:55:31.783010 28336 log.cc:824] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:55:31.783819 28336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1: No bootstrap required, opened a new log
I20260620 14:55:31.785971 28336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } }
I20260620 14:55:31.786108 28336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df243838ef14e098fd3251069ccf4d1, State: Initialized, Role: FOLLOWER
I20260620 14:55:31.786180 28336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } }
I20260620 14:55:31.786525 28336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:55:31.788017 28310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "838c98fa1463461199e4e98d82c671c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2df243838ef14e098fd3251069ccf4d1"
I20260620 14:55:31.787940 28309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:55:31.789140 28309 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [term 1 FOLLOWER]: Refusing update from remote peer 838c98fa1463461199e4e98d82c671c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 14:55:31.789749 28472 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:55:31.789811 28472 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:55:31.790110 28448 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:55:31.791167 28310 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 838c98fa1463461199e4e98d82c671c3 in term 1.
I20260620 14:55:31.792656 28474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "838c98fa1463461199e4e98d82c671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } } }
I20260620 14:55:31.792758 28474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:31.796737 28477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "838c98fa1463461199e4e98d82c671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } } }
I20260620 14:55:31.796826 28477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:31.797472 28461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 838c98fa1463461199e4e98d82c671c3. Latest consensus state: current_term: 1 leader_uuid: "838c98fa1463461199e4e98d82c671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } } }
I20260620 14:55:31.797572 28461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:55:31.807839 28448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "838c98fa1463461199e4e98d82c671c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838c98fa1463461199e4e98d82c671c3" member_type: VOTER last_known_addr { host: "127.5.118.190" port: 37957 } } peers { permanent_uuid: "2df243838ef14e098fd3251069ccf4d1" member_type: VOTER last_known_addr { host: "127.5.118.189" port: 43167 } } peers { permanent_uuid: "e219a0f031484f41933f1881409cdec7" member_type: VOTER last_known_addr { host: "127.5.118.188" port: 44495 } } }
I20260620 14:55:31.807969 28448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:55:31.812335 28450 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:55:31.813225 28450 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:55:31.834455 28450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 838c98fa1463461199e4e98d82c671c3: Generated new TSK 0
I20260620 14:55:31.834707 28450 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:55:31.866978 28265 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:31.865574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.5.118.190:37957)
0620 14:55:31.865622 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.865624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.866257 (+ 633us) server_negotiation.cc:407] Connection header received
0620 14:55:31.866367 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.866368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.866379 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.866421 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.866900 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44496: BlockingRecv error: recv got EOF from 127.0.0.1:44496 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 14:55:31.868256 28416 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:31.865943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58840 (local address 127.5.118.188:44495)
0620 14:55:31.865988 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.865990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.867642 (+ 1652us) server_negotiation.cc:407] Connection header received
0620 14:55:31.867780 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.867781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.867793 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.867822 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.868141 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58840: BlockingRecv error: recv got EOF from 127.0.0.1:58840 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:55:31.869244 28341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:31.867460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49434 (local address 127.5.118.189:43167)
0620 14:55:31.867502 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.867504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.868618 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 14:55:31.868789 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.868790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.868802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.868832 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.869182 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49434: BlockingRecv error: recv got EOF from 127.0.0.1:49434 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:55:31.871874 28341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:31.871010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.5.118.189:43167)
0620 14:55:31.871048 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.871049 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.871295 (+ 246us) server_negotiation.cc:407] Connection header received
0620 14:55:31.871430 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.871431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.871452 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.871484 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.871825 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49440: BlockingRecv error: recv got EOF from 127.0.0.1:49440 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:55:31.875238 28341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:31.874420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.5.118.189:43167)
0620 14:55:31.874457 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.874459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.874697 (+ 238us) server_negotiation.cc:407] Connection header received
0620 14:55:31.874796 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.874797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.874808 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.874836 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.875189 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49450: BlockingRecv error: recv got EOF from 127.0.0.1:49450 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 14:55:31.879501 28341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:31.878704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.5.118.189:43167)
0620 14:55:31.878742 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.878744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.878957 (+ 213us) server_negotiation.cc:407] Connection header received
0620 14:55:31.879063 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.879065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.879076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.879105 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.879454 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49456: BlockingRecv error: recv got EOF from 127.0.0.1:49456 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 14:55:31.885921 28341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:31.885060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.5.118.189:43167)
0620 14:55:31.885102 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.885104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.885341 (+ 237us) server_negotiation.cc:407] Connection header received
0620 14:55:31.885458 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.885459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.885471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.885502 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.885863 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49470: BlockingRecv error: recv got EOF from 127.0.0.1:49470 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 14:55:31.896517 28341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:31.895512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.5.118.189:43167)
0620 14:55:31.895560 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.895562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.895835 (+ 273us) server_negotiation.cc:407] Connection header received
0620 14:55:31.895947 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.895949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.895962 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.895995 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.896454 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49480: BlockingRecv error: recv got EOF from 127.0.0.1:49480 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:55:31.915175 28341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:31.914222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49496 (local address 127.5.118.189:43167)
0620 14:55:31.914271 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.914273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.914563 (+ 290us) server_negotiation.cc:407] Connection header received
0620 14:55:31.914669 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.914671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.914684 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.914715 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.915112 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49496: BlockingRecv error: recv got EOF from 127.0.0.1:49496 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 14:55:31.949800 28341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:31.949003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.5.118.189:43167)
0620 14:55:31.949048 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:31.949051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:31.949136 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:55:31.949194 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:31.949195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:31.949208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:31.949296 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:31.949737 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49500: BlockingRecv error: recv got EOF from 127.0.0.1:49500 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 14:55:32.016434 28341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:32.015606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.5.118.189:43167)
0620 14:55:32.015684 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:32.015687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:32.015696 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:55:32.015723 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:32.015724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:32.015733 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:32.015826 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:32.016372 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49506: BlockingRecv error: recv got EOF from 127.0.0.1:49506 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 14:55:32.146615 28341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:32.145828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.5.118.189:43167)
0620 14:55:32.145881 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:32.145883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:32.146010 (+ 127us) server_negotiation.cc:407] Connection header received
0620 14:55:32.146038 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:32.146039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:32.146052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:32.146106 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:32.146560 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49512: BlockingRecv error: recv got EOF from 127.0.0.1:49512 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 14:55:32.404870 28341 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:55:32.404109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.5.118.189:43167)
0620 14:55:32.404372 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:32.404374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:32.404389 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:55:32.404415 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:32.404416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:32.404426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:32.404479 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:55:32.404818 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49526: BlockingRecv error: recv got EOF from 127.0.0.1:49526 (error 108)
Metrics: {"server-negotiator.queue_time_us":237}
I20260620 14:55:32.739516 28429 catalog_manager.cc:1259] Loaded cluster ID: 9a0a07e9ba794afbbb25e73cc1c53a3e
I20260620 14:55:32.739595 28429 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7: loading cluster ID for follower catalog manager: success
I20260620 14:55:32.740473 28429 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7: acquiring CA information for follower catalog manager: success
I20260620 14:55:32.740749 28429 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e219a0f031484f41933f1881409cdec7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:55:32.790323 28472 catalog_manager.cc:1259] Loaded cluster ID: 9a0a07e9ba794afbbb25e73cc1c53a3e
I20260620 14:55:32.790441 28472 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1: loading cluster ID for follower catalog manager: success
I20260620 14:55:32.791307 28472 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1: acquiring CA information for follower catalog manager: success
I20260620 14:55:32.791646 28472 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2df243838ef14e098fd3251069ccf4d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:55:32.921003 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:32.918144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.5.118.189:43167)
0620 14:55:32.918305 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:32.918307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:32.918320 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:55:32.918476 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:32.918477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:32.918525 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:32.918627 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:32.918788 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:32.919299 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:32.920106 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:32.920332 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:32.920438 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:32.920641 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:32.920759 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:32.920856 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:32.920881 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 14:55:32.920903 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
I20260620 14:55:32.925376 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:32.922785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.5.118.188:44495)
0620 14:55:32.922893 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:32.922895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:32.922902 (+ 7us) server_negotiation.cc:407] Connection header received
0620 14:55:32.922991 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:32.922992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:32.923031 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:32.923110 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:32.923189 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:32.923696 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:32.924563 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:32.924800 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:32.924870 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:32.925064 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:32.925173 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:32.925246 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:32.925283 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 14:55:32.925308 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":45,"threads_started":1}
I20260620 14:55:32.930506 28556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:32.927285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.5.118.190:37957)
0620 14:55:32.927542 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:32.927545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:32.927554 (+ 9us) server_negotiation.cc:407] Connection header received
0620 14:55:32.927603 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:32.927604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:32.927656 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:32.927762 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:32.927911 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:32.928523 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:32.929426 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:32.929700 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:32.929768 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:32.930010 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:32.930135 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:32.930203 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:32.930364 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:55:32.930395 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":45,"threads_started":1}
I20260620 14:55:32.930576 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:32.927304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.5.118.189:43167)
0620 14:55:32.927346 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:32.927347 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:32.927453 (+ 106us) server_negotiation.cc:407] Connection header received
0620 14:55:32.927568 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:32.927569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:32.927582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:32.927613 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:32.927795 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:32.928566 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:32.929728 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:32.930040 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:32.930080 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:32.930272 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:32.930303 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:32.930350 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:32.930458 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 14:55:32.930480 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 14:55:32.930776 28549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:55:32.927526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.5.118.188:44495)
0620 14:55:32.927566 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 14:55:32.927568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 14:55:32.927594 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:55:32.927657 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:55:32.927658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:55:32.927670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:55:32.927718 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:55:32.927860 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:32.928589 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:32.929728 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:55:32.930012 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:55:32.930104 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:55:32.930277 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:55:32.930309 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:55:32.930341 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 14:55:32.930679 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 14:55:32.930708 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 14:55:32.932504 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 28193
I20260620 14:55:32.938565 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 28268
I20260620 14:55:32.943622 5594 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjla6OK/build/release/bin/kudu with pid 28343
2026-06-20T14:55:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2892 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5788 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (279085 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST